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Executive Summary 



In September 2005, the New York City Department of Youth and Community 
Development (DYCD) launched the Out-of-School Time Programs for Youth (OST) initiative to 
provide young people throughout New York City with access to high-quality programming after 
school, on holidays, and during the summer at no cost to their families. Working closely with 
the nonprofit community and with ten City agencies, including the Department of Education 
(DOE), DYCD created a comprehensive public system of out-of-school time programming. 

Each OST program is operated by a nonprofit organization and is located in a school, community 
center, settlement house, religious center, cultural organization, library, or a public housing or 
Parks facility. 

DYCD contracted with Policy Studies Associates (PSA) to conduct a comprehensive 
evaluation of the OST initiative. PSA designed the evaluation to describe the characteristics of 
OST programs and participants and to assess the initiative’s contribution to the growth and 
development of New York City youth. In addition to providing descriptive information on the 
initiative and its participants, the third-year report explores the associations among program 
quality, patterns of youth participation, and youth outcomes, using known features of high- 
quality programs as identified through previous research. 



Scope of Services 

Since its inception in 2005, the OST system has provided services to a total of 181,001 
individual youth throughout New York City. During the 2007-08 school year, more than 81,000 
youth participated in one of 622 OST programs citywide, all of which were available at no cost 
to families. Elementary-grades programs generally provided services from 3 to 6 pm Monday 
through Friday. Middle-grades programs offered at least eight hours of programming per week, 
and high school programs offered a minimum of 108 hours of programming on a flexible 
schedule throughout the school year. In addition, in order to meet the needs of working families, 
programs serving elementary- and middle-grades youth were open on school holidays. Many 
programs also offered programming for 50 hours per week for eight weeks during the summer. 

The OST initiative grew over its first three years to serve increasingly large numbers of 
youth each year. In the first year of the initiative (2005-06), 50,618 youth enrolled in OST 
programs throughout the city, including 40,584 participants in the Option I programs that are at 
the core of this report. In the second year of the initiative (2006-07), this number had increased 
to 68,449 participants overall and 56,742 in Option I, and by the third year the initiative had 
grown to serve 81,213 youth, including 67,524 Option I participants. The initiative also 
expanded in terms of the number of programs offering OST services, with the addition of 1 1 1 
new elementary-grades programs in the third year of the initiative. 
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Program Quality 



Varied program activities. Aligned with core program goals, OST programs generally 
offered varied activities, including academic skills enhancement, cultural exposure and 
enrichment, sports, recreation, community service, and leadership development. Nearly all 
elementary- and middle-grades programs offered academic enhancement, arts and culture, and 
recreational activities. About two-thirds of these programs offered life-skills activities, and 
about half offered community-building activities. Consistent with the age ranges they serve, 
relatively few elementary- and middle-grades programs offered any career or work activities in 
the third year of the OST initiative (15 percent and 24 percent, respectively). In contrast, high 
school OST programs often focused on a narrow set of activities or topic areas. Reflecting that 
focus, the activity patterns in high school programs differed from those of elementary- and 
middle-grades programs: while more than two-thirds of high school programs offered academic 
enhancement, arts and culture, and recreational activities, only about half of high school 
programs offered life skills, community building, and career or work activities. 

Exposure to new experiences. Overall, participants reported a high degree of satisfaction 
with the extent to which their OST program exposed them to new experiences (average score of 
3.20 out of 4 on the youth-survey scale). Elementary-grades youth were somewhat more likely 
to report a high level of participation in new experiences (3.34 out of 4) than were middle-grades 
(3.22) and high school youth (3.27). Based on the evaluation’s statistical measure of the 
difference in responses, high school youth responded notably more positively to this scale than 
did middle-grades youth, suggesting that in general there is most room for improvement in 
creating engaging, innovative activities at the middle-school level. 

Observed program quality. In structured observations of program activities in a 
stratified, random sample of 15 programs, evaluators found that OST staff generally created 
warm, welcoming, yet structured environments. However, programs struggled in their efforts to 
implement active learning opportunities through activities that built on each other in a sequenced 
manner. On average, observed activities demonstrated clear activity goals (5.07, out of 7 on the 
evaluation’s task-oriented scale). Activities also rated high on the relationship-focused scale, 
which measures the extent to which activities developed personal and social skills (4.21, out of 
7). In contrast, evaluators observed less evidence of activity quality as defined by the extent to 
which the activity built on skills and content previously learned (3.60, out of 7, on the skill- 
building scale) or engaged youth in active, hands-on learning experiences (1.75, out of 7, on the 
active-learning scale). 



Positive Relationships Involving Youth 

Youth interactions with peers. Youth were overwhelmingly positive in their assessment 
of their social interactions within the program (average scale score of 3.32, out of 4), although 
both middle-grades and high school youth responded somewhat more positively than did 
elementary-grades participants (3.35 and 3.37, compared to 3.28). These high scores are an 
indication that OST programs were successful in creating welcoming social environments for 
youth to develop friendships. 
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Youth interactions with staff. OST participants also responded very positively to a scale 
measuring the quality of their interactions with program staff (average scale sore of 3.35, out of 
4), with high school youth responding somewhat more positively than either elementary-grades 
or middle-grades youth (3.46, compared to 3.31 for both elementary- and middle-grades youth). 



Effective Program Partnerships and Supports 

Staffing patterns. The majority of programs (82 percent) hired college students, and 69 
percent of programs hired at least one professional specialist (e.g., a professional artist, coach, 
dancer). In addition, 63 percent of programs had at least one certified teacher on staff, and 61 
percent hired teen staff members. Fifty percent of OST program directors reported that a staff 
member was assigned to be a master teacher or educational coordinator within the program. 

OST programs were strategic in the roles they assigned to certified teachers and 
specialists within the programs. Certified teachers were employed mainly to provide academic 
support to programs by leading academic activities (72 percent of programs) and tutoring (7 1 
percent). Specialists were hired primarily for non-academic enrichment activities such as arts 
and sports (88 percent). College students, in contrast, played roles across program activities, 
including tutoring youth (88 percent) and assisting with enrichment and academic activities (82 
percent and 78 percent, respectively). More than two-thirds of programs also employed college 
students to lead enrichment and academic activities (73 percent and 70 percent, respectively). 
Teen staff were mostly hired as activity assistants (82 percent) or as tutors (80 percent). Through 
these mixed staffing patterns, programs aimed to involve young staff members in connecting 
with younger participants; teen staff were supported by experienced staff who could provide 
ongoing mentoring and supervision. Thanks to these varied staffing patterns, including the use 
of both paid and volunteer staff, OST programs maintained an overall low ratio of youth-to-staff, 
with a median ratio of 8: 1 . 

Building staff capacity. With the launch of the OST initiative in 2005, DYCD contracted 
with the Partnership for After-School Education (PASE) to provide technical assistance and 
professional development workshops for OST program staff. In survey responses, 88 percent of 
program directors reported that PASE was a primary source of technical assistance and training 
for themselves and their staff members. Other primary sources of technical assistance included 
the provider organizations (67 percent) and the DYCD program manager (40 percent). Program 
directors reported that their staff received professional development through staff meetings at the 
program (86 percent), internal staff orientations (66 percent), and off-site workshops (62 
percent). 

Partnerships with schools. In the third year of the OST initiative, program directors 
reported regular communication with school staff in several areas. Program directors reported 
communicating with school administrators or staff at least monthly about: the needs or progress 
of individual students (61 percent), issues related to sharing classroom space (56 percent), 
homework assignments (56 percent), and student discipline policies (50 percent). In surveys, 
directors reported high levels of satisfaction with their communication with school staff. Sixty- 
one percent of program directors said that receiving responses to requests to coordinate services 




or resources with school staff was not a challenge; only 6 percent reported that this was a major 
challenge. 



Partnerships with parents. Programs reached out to the families of participants to 
engage them and to meet youth needs more effectively. Nearly all program directors (91 
percent) had conversations with parents over the phone at least a few times a month; 83 percent 
met in person with parents that frequently. Programs also relied on family or parent liaisons to 
engage families and encourage high rates of participation: 45 percent of programs employed 
someone for this role. 



Evidence of Youth Outcomes 

Program engagement. In the third year of the OST initiative, programs on average 
exceeded their targeted enrollment levels. Option I programs had a target enrollment overall of 
approximately 63,000 youth, based on the contracts awarded by DYCD; programs actually 
served a total of about 64,500 students from September 2007 through June 2008. Even so, some 
individual programs could not meet their targeted enrollment, measured by the number of slots 
available for participants as established in the program’s contract with DYCD. Sixty-nine 
percent of elementary programs met or exceeded their enrollment targets, as did 70 percent of 
middle-grades programs and 59 percent of high school programs. 

As in Years 1 and 2, programs reached a high standard of participation. On average, 
elementary-grades participants attended 377 hours during the year, compared to the 432 hours 
they were expected to attend. This represents an average of 87 percent of targeted hours, 
exceeding average participation rates of 72 percent and 83 percent attained in Year 1 and 2, 
respectively. Older youth also surpassed their targeted number of OST participation hours, on 
average. Middle-grades participants as a group achieved their targeted number of hours of 
participation: on average, middle-grades participants attended 218 hours of the 216 hours 
expected at the middle-grades level (101 percent of the targeted hours). High school participants 
exceeded their targeted number of hours of participation, attending on average 92 hours in the 
third year of the initiative, 16 hours above their target of 76 hours. 

The evaluation also measured the duration of participation across years: 6,371 youth 
participated in all three years of OST school-year programming. Almost 22,000 youth 
participated in two years of OST programming, either in Years 1 and 2 or in Years 2 and 3, while 
102,837 participated in a single year of programming (Year 1, Year 2 or Year 3). Approximately 
22,000 youth only participated in summer programming. 

Social development. Overall, participants reported a strong sense of belonging in the 
third year of the initiative, with an average youth-survey scale score of 3.38, out of 4. In 
particular, more than two-thirds of participants “agreed a lot” that they felt safe in the program 
(68 percent) and 58 percent said that the program was a “good place to hang out.” High school 
students were especially likely to report a strong sense of program connection (scale score of 
3.48, compared to 3.32 for elementary-grades participants and 3.38 for middle-grades 
participants). 
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Educational development. Analyses examined the relationship between OST 
participation and the following measures of educational development: academic motivation, 
school attendance rate, academic benefits reported by youth, performance on the state English 
language arts (ELA) and mathematics tests (for grades 3-8), and credits accrued and Regent 
exams passed for high school students. 

As in prior years, participants reported an overall strong level of academic motivation in 
Year 3, with an average scale score of 3.34 out of 4. Elementary-grades participants reported 
notably higher levels of academic motivation than did middle-grades or high school participants 
(scale score of 3.34, compared to 3.25 for both groups of older youth). 

Participants reported moderate levels of academic benefits on a survey scale (average 
scale score of 3.06, out of 4). The most common academic benefit reported by participants was 
that the program helped them to finish their homework more often (54 percent agreed a lot). 
However, no significant differences were found between participants and matched 
nonparticipants on the measures of educational perfonnance that are maintained by the 
Department of Education. OST participants and nonparticipants in the 15 sampled sites were 
closely matched on their baseline academic perfonnance in both the state ELA and math tests. 
Consistent with citywide trends, both groups showed small improvements in performance over 
the course of the OST initiative, with no significant differences in the size of the gains posted by 
the two groups. For high school youth in the sampled sites, evaluators examined the cumulative 
number of course credits accrued after each year of OST participation. Analyses found no 
significant differences between the groups on this measure. Analyses also examined the number 
of New York State Regents exams that participants and matched nonparticipants had taken and 
passed to assess progress towards graduation, and found no significant differences. Analyses of 
school-attendance patterns of participants and matched nonparticipants found no notable 
differences in attendance changes over time. 



Relationships Among Participation, Program Quality, and Youth Outcomes 

Relationship between program quality and youth outcomes. Overall, correlation 
analyses revealed positive relationships between youth reports of the extent to which a program 
exposed youth to new experiences and youth reports of their sense of belonging in the OST 
program, their academic benefits, and their rates of school attendance. However, a program’s 
mean rating of exposing youth to new experiences was negatively correlated with individual 
youths’ mean hours of participation in OST. A possible explanation for this pattern is that youth 
who attend OST more become accustomed to the program offerings over time and therefore 
lower their perceptions of novelty within the program. The breadth of OST program content in 
OST programs, measured by the number of different activity types offered, was positively 
associated with the total number of hours of youth participation and with the number of credits 
earned by high school participants. 

Relationships involving social interactions. In general, measures of a supportive OST 
environment, including average reports of youth interactions with their peers and with staff 
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members in the program, were positively correlated with youth reports of their sense of 
belonging and of the program’s academic benefits. 

Relationships involving effective partnerships and supports. Several measures of 
effective partnerships and supports were negatively associated with youth outcomes. For 
example, the number of professional development opportunities in which OST staff members 
participated was negatively correlated with youth program participation, youth reports of sense 
of belonging, and school attendance. Rather than being an indication that staff professional 
development is ineffective, more likely this finding is an indication that programs that are 
struggling to implement a high-quality program and contribute to positive youth outcomes are, in 
fact, taking greatest advantage of the professional development opportunities available through 
DYCD, PASE, and other resources, either as a result of a referral to technical assistance by their 
DYCD program manager or by their own choice. Measures of communication with schools and 
with parents were positively associated with the number of hours of youth participation in OST 
programming. 

Program quality index. Based on all of the findings from the correlation analyses, 
evaluators created a program quality index as a tool for assessing the overall quality of an OST 
program. Final components of the program quality index include exposure to new experiences, 
youth interactions with peers, youth interactions with staff, wide mix of staff, presence of a 
master teacher, and presence of a parent liaison. Analyses found a positive relationship between 
the program quality index and whether the program succeeded in meeting its targeted enrollment 
level. Analyses also found positive correlations between overall program quality and aggregate 
youth reports of their sense of belonging in the program, their engagement in pro-social 
behaviors, their academic motivation, and their academic benefits. Multi-level analyses revealed 
some significant relationships between OST program quality and youth outcomes, even though 
program-level variables accounted for only a small amount of variance in youth outcomes. 



Creation of a City-wide OST System 

Capacity of provider organizations. Executive directors of provider organizations 
reported that the OST initiative increased their organizational capacity in several ways. More 
than half of executive directors reported that the initiative increased the organization’s capacity 
to reach out to serve more youth and families to a great extent or somewhat (83 percent), provide 
staff training and technical assistance (73 percent), partner with a public school (71 percent), 
partner with cultural organizations (65 percent), partner with city agencies (63 percent), offer 
programming on weekends and holidays (59 percent), and provide a career ladder for OST staff 
(57 percent). 

Executive directors also reported increased capacity to implement certain practices in the 
third year of the OST initiative, compared to previous years. Fifty-nine percent of executive 
directors reported that their programs provided much more or somewhat more training and 
technical assistance for staff in the third year of the initiative than in the first or second year. 
Overall, executive directors reported high levels of satisfaction with DYCD’s management of the 
OST initiative and support of OST. More than half of executive directors reported that they were 
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very satisfied with the support provided by the DYCD program manager (67 percent) and the 
opportunities for staff professional development (55 percent). Executive directors were least 
satisfied with DYCD’s assistance in negotiating partnerships with schools (29 percent reported 
that they were very satisfied). 

Meeting the needs of working families. A goal of the OST initiative was to provide 
support to working families in New York City, particularly in the target zip codes identified as 
priorities for out-of-school time services. Survey responses from parents of OST participants in 
the elementary- and middle-grades indicated that overall the initiative succeeded in reaching this 
goal and meeting the needs of families: about three-quarters of parents rated the OST program 
that their child attended as either excellent (43 percent) or very good (33 percent). 

Parents especially valued the academic-support features of OST programs. Forty-seven 
percent of parents cited homework help as the most important activity in the after-school 
program, and an additional 26 percent cited academic enrichment as the most important activity. 
In addition, parents’ reports of their reasons for enrolling their child in the OST program 
reflected an emphasis on seeking academic support: 76 percent believed the program would help 
their child do better in school, and 72 percent wanted their child to get help with homework. 
Seventy-five percent of parents also said that they enrolled their child in an OST program to 
provide them with the opportunity to participate in new activities. 

Parents also responded positively to questions about the ways in which the OST initiative 
had enabled them to work more or pursue more education. Across all responding parents, 74 
percent agreed that the program made it easier for them to keep their job, and 73 percent agreed 
that they miss less work than they had previously because their children attended the OST 
program. In addition, 7 1 percent of parents reported that they were able to work more hours 
because their children were in the program. 



Looking Ahead to Long-Term Sustainability 

Through the first three years of the OST initiative, DYCD has established foundations in 
policy and practice for the long-term sustainability of high-quality, publicly funded, out-of-school 
time programming for the youth of New York City. With the launch of the OST initiative in 2005, 
the collaborative relationship between DYCD and the New York City Department of Education 
has grown, particularly through adoption of a Memorandum of Understanding committing 
hundreds of public schools as sites for OST programs and the provision of in-kind support by the 
DOE for OST programs through facilities, security, and snacks and meals for OST participants. At 
the same time, DYCD strengthened its network of community-based organizations, foundations, 
and providers of technical assistance to support the initiative through partnerships with the Wallace 
Foundation and the Partnership for After-School Education, among others. Importantly, the City 
has included funding for OST programming in its four-year financial plan and the budget for OST 
programming has steadily increased from $46.4 million in FY 2006 to $1 17.1 million in FY 2009. 

As the OST initiative enters its next phase, evaluation findings from the first three years 
point to elements of program quality to maintain as well as possible areas in which focused 
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resources and technical assistance can improve quality and youth outcomes. DYCD has already 
committed resources to improving the quality of its monitoring and support of OST programs 
through ongoing technical assistance opportunities, particularly those focused on data 
management, behavior management techniques, and program content development. The DY CD 
Online data management system offers opportunities to continue to track patterns of program and 
activity engagement across the initiative and within specific programs. The development of new 
program management tools, including those that track the implementation of features in the 
evaluation’s program quality index, will provide opportunities to continue to strengthen the 
capacity of OST programs to provide high-quality services to youth. 

Additional research-based steps will support the city’s OST providers as they shape the 
city’s OST system of the future. These steps include the following: 

■ Assist programs in identifying resources — or in learning to better plan and budget 
existing resources — directed specifically to hiring specialized staff members to 
maximize youth recruitment and engagement (e.g., parent liaisons) and to help 
plan and oversee high-quality, structured program content (e.g., certified teachers 
or professional specialists). 

■ Focus technical assistance related to activity planning on teaching staff strategies 
to engage youth in dynamic, active learning opportunities in which they discuss, 
collaborate, plan, and take on leadership roles, regardless of the content area. 

■ Through technical assistance, encourage OST programs and provider 
organizations to utilize the capacity of the DYCD Online system to generate data 
that can support program management and improvement efforts, including, for 
example, the monitoring of program participation patterns to determine whether 
certain types of activities appeal more or less to particular groups of students. 
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1. Overview of the OST Initiative and This Evaluation 



In September 2005, the New York City Department of Youth and Community 
Development (DYCD) launched the Out-of-School Time Programs for Youth (OST) initiative to 
provide young people throughout New York City with access to high-quality programming after 
school, on holidays, and during the summer at no cost to their families. Working closely with 
the nonprofit community and with ten City agencies, including the Department of Education 
(DOE), DYCD created a comprehensive public system of out-of-school time programming. 

Each OST program is operated by a nonprofit organization and is located in a school, community 
center, settlement house, religious center, cultural organization, library, or a public housing or 
Parks facility. 

DYCD designed the OST initiative to deliver high-quality OST services under three 
program options. Option I, which is the focus of this report, supports comprehensive OST 
programs throughout New York City. Program expectations for Option I vary by grade level, 
with programs for younger youth expected to provide more programming hours and hence more 
comprehensive services to youth who attend programs on a frequent basis. Option II was 
designed to support OST programs that use private matching funds to subsidize at least 30 
percent of their OST budgets, and Option III programs were to be operated in collaboration with 
the Department of Parks and Recreation at Parks sites. All options were designed to serve youth 
of all grade levels. 

The OST initiative concentrates services in high-need neighborhoods, targeting public 
funding to high-priority zip codes identified based on the size of the youth population, the youth 
poverty rate, the percent of youth disconnected from school or work, the number of English 
Language Learners in public schools, the number of single-parent families, and the number of 
children eligible for state-subsidized childcare. 

Since its launch in 2005, New York City’s investment in the OST initiative has grown. 
The OST budget increased from $46.4 million in FY 2006, the first year of the initiative to $76.8 
million in FY 2007 and $1 17.1 million in FY 2009. Funding for the OST initiative is included in 
the City’s four-year financial plan and is thus expected to provide sustainable out-of-school time 
services for New York youth into the future. In addition to public funds, the City has received a 
$12 million grant from the Wallace Foundation to support system-wide OST development. 

DYCD contracted with Policy Studies Associates (PSA) in 2005 to conduct a 
comprehensive evaluation of the OST initiative. PSA designed the evaluation to describe the 
characteristics of OST programs and participants and to assess the initiative’s contribution to the 
growth and development of New York City youth. The Year 2 evaluation report (Russell, 
Mielke, & Reisner, 2008) examined evidence of OST programs’ achievement of high quality and 
broad scale. Evaluators analyzed the extent to which programs were establishing structures to 
support high-quality programming, successfully engaging youth, and developing youth skills in 
both social and educational domains. 
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Among other findings of the Y ear 2 evaluation report, evaluators identified eight program 
quality features that were positively associated with high levels of program participation and 
with desirable social and academic outcomes. 1 The quality features were consistent with recent 
research on out-of-school time programming (Durlak & Weissberg, 2007; Grossman, Campbell, 
& Raley, 2007; Little, 2007) and with the elements of positive developmental settings identified 
by the National Research Council’s Committee on Community-Level Programs for Youth 
(Eccles & Gootman [Eds.], 2002). The quality features include the following: 

Rich Program Content 

■ Programs offer a variety of both academic enrichment and non-academic 
activities, including arts, recreation, and civic engagement. 

■ Youth are exposed to new and engaging experiences. 

Positive Relationships 

■ Y outh have opportunities to interact with their peers. 

■ Youth interact with and develop positive relationships with staff. 

Effective Partnerships and Supports 

■ Programs staffing patterns include younger staff members, who are supported by 
more experienced staff. 

■ Program directors and staff participate regularly in professional development. 

■ Programs communicate with schools regularly about learning objectives and 
methods. 

■ Programs reach out and engage families through parent liaisons and special events 
for parents. 

This report explores the associations among OST program quality, youth participation, 
and youth outcomes, using the preceding program features as the guiding framework. The report 
first describes the extent to which these elements of program quality are present in OST 
programs. It next summarizes evidence regarding youth engagement and social and educational 
outcomes for participants in the OST initiative. The report then reviews analyses examining the 
relationships among youth social and educational outcomes, level of youth participation, and 
program quality. Finally, the report describes the extent to which the OST initiative has helped 
to build a system for comprehensive out-of-school time services in New York City, including 



1 The Year 2 evaluation report also identified a ninth programmatic feature associated with program quality — the 
opportunity for youth to participate in both summer and school-year programming. However, this feature depends 
on program funding and may not be under programs’ control to implement. For this reason, it was not included as a 
key element in program-quality analyses in Year 3. 
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development of the capacity of provider organizations to deliver high-quality services and meet 
needs of working families. 

Findings in this report are based on data collected during Years 1 through 3 of the OST 
initiative (2005-06 through 2007-08) from the sources listed below. Descriptive analyses of 
program quality and of systems-building focus on Year 3 data, noting when changes occurred 
over the course of the initiative. Analyses of youth outcomes examine data from across the first 
three years of the OST initiative, describing youth outcomes after one year, two years, and three 
years of participation in OST programming. Detailed summaries of the data collected in each 
year are presented in Appendix A. The report focuses on programs funded under Option I of the 
OST initiative; analyses of data from Option II and III programs are presented in Appendix B. 
Specific data sources include the following: 

■ OST enrollment, participation, and activity data, as collected in DYCD Online, 
the agency’s participant tracking system 

■ Survey of executive directors of all OST provider organizations 

■ Survey of program directors overseeing the day-to-day operations of all OST 
programs 

■ Survey of OST participants in grades 3-12 who attended any of the 133 
randomly selected Option I programs in the evaluation sample from which 
evaluators received appropriate principal and parental consents to participate in 
the evaluation 

This sample was structured to be representative of Option I OST programs as a 
whole, including their distribution by grade level served and by whether the 
program was school-based or center-based. 

■ Demographic, school attendance, and educational performance data from DOE’s 
student records for participants with parental research consent who attended one 

of the evaluation’s 15 in-depth Option I programs at any time during the initiative, 

2 

and a group of matched nonparticipants 

■ Survey of program staff members in the 1 5 randomly selected in-depth study 
programs 

■ Survey of parents of OST participants in the 15 in-depth study programs 

■ Interview and observation data collected during site visits to each of the in- 
depth study programs 



2 These 15 OST programs were randomly selected from within the larger participant-survey sample, stratified to be 
representative based on grade level served and on whether the program was located in a school or other site. 
However, the sample was not designed to be able to detect differences among groups for small program effects. 
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These visits included interviews with the program director, program staff, 
participants, and, in many cases, parents. Evaluators also conducted structured 
activity observations during these visits. 



Analytic Approach 

All findings presented in the report are statistically significant at the p<0.05 level. In 
addition, for each comparative or associative measure reported, the evaluation computed an 
effect size to measure the magnitude or strength of the finding. 

The statistical tests and measures of effect size used in analysis varied based on the 
properties of the data analyzed. For analyses of continuous variables, the evaluation generally 
selected an independent samples t-test and computed a Cohen’s cl measure of effect. For 
categorical data, the evaluation conducted chi-square analyses and reported either a Cramer’s V 
effect (for nominal data) or a gamma (y) statistic (for ordinal data). Analyses of association 
between continuous variables typically relied on a Pearson’s correlation (r). For analyses of 
participant survey scales and observational data, data typically were not normally distributed. In 
those cases, the evaluation employed nonparametric tests, including the Mann- Whitney U test as 
an alternative to the independent samples t-test, a Kruskal-Wallis test as an alternative to the 
ANOVA, and a Speannan correlation as the nonparametric alternative to the Pearson’s 
correlation. The effect sizes for the Mann- Whitney U test and the Kruskal-Wallis test are both 
reported as tes, for the Spearman correlation, r s serves as the effect size measure. For 
correlations with dichotomous variables, a point-biserial correlation coefficient (r p /,) is reported. 

Conventions for educational research suggest that effect size values between 0.10 and 
0.20 indicate a “small but meaningful” association, between 0.21 and 0.50 an “important” 
association, and 0.51 or higher an “impressive” association (Cohen, 1988; Lipsey, 1990). This 
report focuses on findings with an effect size of at least 0.10; comparisons or associations below 
this threshold were considered too weak to warrant reporting. In general, however, while the 
associations discussed in this report describe notable relationships between program structures 
and youth outcomes, they should not in any instance be interpreted as implying causation. 



4 




2. Scope of OST Services 



Since its inception in 2005, the OST system has provided services to a total of 181,001 
individual youth (an unduplicated count) across New York City and across Options I, II, and III. 
During the 2007-08 school year alone, more than 81,000 youth participated in one of 622 OST 
programs city wide, all of which were all available at no cost to families. Elementary-grades 
programs generally provide services from 3 to 6 pm Monday through Friday. Middle-grades 
programs offer at least eight hours of programming per week, and high school programs offer a 
minimum of 108 hours of programming on a flexible schedule throughout the school year. In 
addition, in order to meet the needs of working families, programs serving elementary- and 
middle-grades youth are open on school holidays. Many programs also offer OST-supported 
programming for eight weeks during the summer. 

As shown in Exhibit 1, the OST initiative grew over its first three years to serve 
increasingly large numbers of youth. In the first year of the initiative, a total of 50,618 youth 
enrolled in OST programs throughout the city, including 40,584 participants in the Option I 
programs that are the focus of this report. In the second year of the initiative (2006-07), this 
number had increased to 68,449 participants overall and 56,742 in Option I, and by the third year 
the initiative had grown to serve 81,213 youth, including 67,524 Option I participants. The 
initiative also increased the number of programs offering OST services, with the addition of 1 1 1 
new elementary-grades programs in the third year of the initiative. 



Exhibit 1 

Total OST Enrollment Across Years 



Component 


Year 1 


Summer 06 


Year 2 


Summer 07 


Year 3 


Summer 08 
(July only) 


Overall 


50,618 


13,160 


68,449 


16,257 


81,213 


29,757 


Option 1 


40,584 


10,140 


56,742 


16,257 


67,524 


26,761 


Option II 


9,024 


3,020 


10,448 


n/a* 


12,340 


2,996 


Option III 


1,010 


0 


1,259 


n/a* 


1,349 


n/a 



*Youth enrollment in summer programs is included in school-year figures. 



Exhibit reads: Across all OST initiative options, 50,618 participants were enrolled in Year 1. 



The OST initiative has served youth of diverse backgrounds from across the five 
boroughs, as shown in Exhibit 2. Most youth participants attended OST programs located in 
public schools (79 percent); the remaining youth attended OST programs in center- or 
community-based locations. More than half of the youth served were in the elementary grades 
(58 percent); 23 percent were in the middle grades; and 18 percent were in high school. As 
shown in Exhibit 3, across all Option I programs, slightly more girls than boys enrolled in 
programming (51 percent, compared to 49 percent), and more than three-quarters of participants 
were either African American or Hispanic/Latino (39 percent and 38 percent, respectively). 

Most participants (84 percent) were eligible for free or reduced-price lunch, and 20 percent were 
English Language Learners. 
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Exhibit 2 

Numbers of Option I OST Programs and Participants in Year 3 



Program 

Characteristics 


Programs 

n=514 


Participants 

n=67,524 


Borough 






Brooklyn 


188 


22,208 (33) 


Bronx 


108 


17,141 (25) 


Manhattan 


85 


10,994 (16) 


Queens 


107 


13,979 (21) 


Staten Island 


26 


3,202 (5) 


Program Location 






School 


369 


53,565 (79) 


Center 


145 


13,959 (21) 


Grade Level Served 






Elementary 


284 


39,376 (58) 


Middle 


116 


15,764 (23) 


High 


114 


12,384(18) 



Figures in parentheses are the percent of participants with each characteristic within each category. 
Percents do not necessarily add to 100 due to rounding. 

Exhibit reads: In Option 1 programs in Year 3, there were 188 programs in Brooklyn. These 
programs served 22,208 youth, who constituted 33 percent of all New York City participants. 



Exhibit 3 

Demographic Characteristics of Participants in Year 3, in Percents 



Gender 


n=67,524 


Male 


49 


Female 


51 


Race/ethnicity 


n=67,524 


American Indian 


1 


Asian 


9 


African American 


39 


Hispanic/Latino 


38 


Pacific Islander 


0 


White (non-Hispanic) 


7 


Other 


6 


Educational characteristics 


n= 1,880 


Eligible for free/reduced price lunch 


84 


English Language Learner 


20 


Eligible for special education services 


16 



Sources: DYCD Online (gender and race/ethnicity); DOE (educational characteristics) 

Exhibit reads: Forty-nine percent of participants in Year 3 were male, 
and 51 percent were female. 
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3. Evidence of Program Quality in Year 3 



The vision for OST described by DYCD in its 2005 RFP called for: the provision of safe 
and developmentally appropriate environments for youth; support for their academic, civic, 
creative, social, physical, and emotional development; and response to the needs of New York 
City’s families and communities. The objectives reported by OST program directors in surveys 
consistently reflect these diverse goals. In Year 3, nearly all program directors (98 percent) 
reported that providing a safe environment for youth was a major objective of their program. In 
addition, more than three-quarters of program directors cited the following as major objectives of 
their programs, indicating a high priority on social development goals: 

■ Help youth develop socially (93 percent) 

■ Promote respect for diversity (91 percent) 

■ Provide health/well-being/life skills development (85 percent) 

■ Provide opportunities for cultural enrichment (85 percent) 

■ Provide recreational activities (84 percent) 

■ Help youth improve their academic performance (80 percent) 

■ Support working families (80 percent) 

■ Provide hands-on academic enrichment activities (78 percent) 

■ Provide leadership opportunities for youth (78 percent) 

Continuing a trend noted in the second year, the percent of program directors who reported 
that academic improvement was a major objective of their program declined relative to reports in 
the prior year. In Year 3, as in Year 2, 80 percent of program directors reported that academic 
improvement was a major objective, compared to 88 percent in Year 1 (V=.10). In contrast, in 
Year 3 evaluators found a substantial increase in the percent of program directors who identified 
support for working families as a major objective, compared to Year 1 and Year 2 of the initiative, 
perhaps reflecting the challenging economic circumstances that emerged in Year 3 (80 percent, 
compared to 71 percent in Year 1 and 69 percent in Year 2; V=.13). 

The remainder of this chapter reviews evidence of program quality as organized in three 
clusters of program features, including rich program content, positive relationships, and effective 
partnerships and supports. 



Rich Program Content 

Research on youth development and on teaching and learning underscores the importance 
of providing youth with rich content-based experiences, led by instructors or coaches who 
encourage mastery of both skills and knowledge and who use both structured and unstructured 
teaching strategies that capture youth interest (Bransford, Brown, & Cocking, 1999; McLaughlin, 
2000). Evaluators used survey data, activity observation data, and program reports in DYCD 



; All comparative or associative findings presented in the report text are statistically significant at the p<0.05 level. 
For each of these findings, evaluators also computed an effect size, using methods that varied based on the 
properties of the data analyzed, as described more fully on page 4. 
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Online to examine the extent to which OST programs provided engaging, enlightening program 
experiences to youth. 



Varied Academic and Non-academic Program Activities 

Aligned with core program goals, OST programs generally offer varied activities, 
including academic skills enhancement, cultural exposure and enrichment, sports, recreation, 
community service, and leadership development. Evaluators assessed the mix of activities in 
each OST program, using DYCD Online records of youth participation in specific activities. 
Program staff categorize each actual OST activity under one of the 17 activity categories 
established in DYCD Online. For analysis purposes, evaluators collapsed these categories into 
six primary activity types, including academic enhancement, career and work, life skills, 
community building, arts and culture, and recreation. 4 

As shown in Exhibit 4, almost all elementary- and middle-grades programs offered 
academic enhancement, arts and culture, and recreational activities (ranging from 92 percent to 
100 percent of programs). About two-thirds of these programs offered life-skills activities, and 
about half offered community-building activities. Consistent with the age ranges they serve, 
relatively few elementary- and middle-grades programs offered any career and work activities in 
the third year of the OST initiative (15 percent and 24 percent, respectively). As reported in 
Russell, Vile, Reisner, Simko, Mielke, & Pechman (2008), high school OST programs often 
were structured to focus on a targeted set of activities or topic areas. Reflecting that focus, the 
activity patterns in high school programs differed from those of elementary- and middle-grades 
programs: more than two-thirds of high school programs offered academic enhancement, arts 
and culture, and recreational activities, and about half of high school programs offered life skills, 
community building, and career and work activities. 

Exhibit 4 

Percent of Programs Offering Types of Activities, by Grade Level 





Elementary 


Middle 


High 


Activity Category 


n=284 


n=1 16 


n=1 14 


Academic Enhancement 


100 


99 


84 


Arts and Culture 


94 


97 


68 


Recreation 


92 


97 


68 


Life Skills 


64 


67 


57 


Community Building 


51 


48 


52 


Career and Work 


15 


24 


50 



Exhibit reads: One hundred percent of elementary programs offered some type of academic 
enhancement activity, as did 99 percent of middle-grades and 84 percent of high school programs. 



4 Evaluators excluded the following activity types from categorization: snack/supper. DOE extended day, DOE 
summer school, and holiday programming. 
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Exhibits 5-7 illustrate the extent or 
intensity of activity offerings, as measured 
by the proportion of total programming time 
focused on each activity. These exhibits 
indicate that this proportion of time devoted 
to each activity varied by grade level, with 
elementary- and middle-grades programs 
delivering a greater concentration of 
academic enhancement, arts and culture, and 
recreation activities, in comparison to 
programs serving older youth. High school 
programs showed a wider range of program 
offerings. On average, 46 percent of the 
hours of programming offered by 
elementary programs was devoted to 
academic enhancement activities, followed 
by 20 percent in arts and culture and 19 
percent in recreation. Middle-grades 
programs, on average, spent about a third of 
programming time on academic enhancement activities (35 percent) and recreation activities (27 
percent). Across the initiative, high school programs spent, on average, 30 percent of their total 
programming hours on academic enhancement activities, followed by recreation (23 percent of 
their hours). High school programs also spent 17 percent of their programming hours on arts and 
culture activities, 1 1 percent each on life skills and on career and work activities, and 8 percent 
on community building. 



Example of an Academic Enhancement Activity 

In a middle-grades program, evaluators observed the 
rehearsal of a spoken word poetry event. Youth 
practiced performing poems they had written, and 
they received feedback from the staff and youth 
audience. The activity observed was participants’ 
first chance to perform in front of a crowd; youth did 
not yet have their poems memorized and read from a 
computer screen. Youth took turns performing the 
poetry they had written, and the staff opened up the 
room for discussion and feedback after each 
performance. Overall, the poems reflected 
participants’ developing skills and employed 
complicated rhyming and wordplay. Staff gave 
constructive criticism and support to students who 
performed: “Go to the place in your heart or mind 
that you were at when you wrote the poem.” A staff 
member modeled performing and receiving criticism 
by reading his own poem and responding to the 
group’s reactions and suggestions. 



Exhibit 5 

Proportion of Hours Offered by Activity Type, Elementary-Grades Programs, 

in Percents (n=284) 




Exhibit reads: On average, 46 percent of the hours that elementary-grades programs offered were in academic 
enhancement activities. 
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Exhibit 6 

Proportion of Hours Offered by Activity Type, Middle-Grades Programs, 

in Percents (n=116) 



Recreation 

27 % 



Community Building Career and Work 

3 % \ / 1 % 

Life Skills \ S' 

(I) 



Academic 

Enhancement 

35 % 



Arts and Culture 
22% 



Exhibit reads: On average, 35 percent of activity hours that middle-grades programs offered were in academic 
enhancement. 



Exhibit 7 

Proportion of Hours Offered by Activity Type, High School Programs, 

in Percents (n=1 14) 




Exhibit reads: On average, 30 percent of the hours that high school programs offered were in academic enhancement 
activities. 
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Exposure to New and Engaging Experiences 

To measure the extent to which participants reported that their OST program exposed 
them to new and engaging activities, evaluators asked a series of questions, and participants were 
asked whether they agreed a lot, agreed a little, disagreed a little, or disagreed a lot with each of 
the statements in the question. As shown in Exhibit 8, participants were generally positive in 
their responses to individual survey questions. For example, 52 percent agreed a lot that the 
activities in their OST program really get them interested and 34 percent agreed a little. Only 14 
percent of participants disagreed with this statement. 

Exhibit 8 

Participant Reports of Exposure to New Experiences, in Percents (n=6, 186) 



In this program... 



The activities really get me interested 



I do a lot of new things 



There is a lot for me to choose to do 



I work on projects that really make me think 



I do things I don't usually get to do anywhere else 



■Agree a lot nAgree a little 

Exhibit reads: Fifty-two percent of participants surveyed agreed a lot that the activities in their program 
really get them interested and 34 percent agreed a little. 

The individual survey items were then combined into a scale calculated to range from 1 
to 4, with 4 indicating that on average participants agreed a lot with each of these statements, and 
1 indicating that on average participants disagreed a lot with each of these statements. Technical 
details about the range and reliability of all youth survey scales are included in Appendix C. 

Similar to previous years, participants on average reported a high degree of satisfaction 
with the extent to which their OST program exposed them to new experiences (average score of 
3.20 out of 4 on the survey scale). Overall, elementary-grades youth were somewhat more likely 
to report a high level of exposure to new experiences (3.34) than were middle-grades (3.22; d=. 12) 
and high school youth (3.27; d=. 20). High school youth also responded notably more positively to 
this scale than did middle-grades youth (d=.15), suggesting that in general there is most room for 
improvement in creating engaging, innovative activities at the middle-grades level. 

In addition to the activity content offered, content delivery strategies contribute to 
program quality and can vary greatly from one program to the next. Using an out-of-school time 
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program observation scale developed by PSA researchers (Pechman, Mielke, Russell, White, & 
Cooc, 2008), evaluators observed program activities in the 12 elementary- and middle-grades 
programs visited in the third year of the initiative. They conducted structured observations and 
rated the quality of program activities on four scales measuring the content delivery and youth 
program experience. The scales, which are described in Exhibit 9, are grounded in recent 
research on out-of-school time quality (Durlak & Weissberg, 2007). Technical details about 
these observational scales are presented in Appendix D. 

Exhibit 9 

Results of Observational Scales (n=141) 



Skill-Building 



Mean 



Alpha 



Activity builds on and expands skills and content learned to 
increase youth knowledge and understanding 



3.60 out of 7 



0.88 



Activity involves the practice or a progression of skills 

Staff challenge youth to move beyond their current level of competency 

Activity requires analytic thinking 

Staff employ varied teaching strategies 

Activity challenges students intellectually, creatively, developmental^, and/or physically 

Staff assist youth without taking control 

Staff verbally recognize youth efforts and accomplishments 



Active Learning 



Mean 



Alpha 



Activity offers youth opportunities to actively participate in 
learning 



1.75 out of 7 



0.67 



Staff plan for and ask youth to work together 
Youth are collaborative 

Youth take leadership responsibilities and roles 
Youth have opportunities to make meaningful choices 
Youth assist one another 

Youth contribute opinions, ideas, and concerns to discussions 
Staff encourage youth to share their ideas, opinions, and concerns 
Staff ask youth to expand on their answers and ideas 



Relationship-Focused 



The activity focuses on developing positive relationships 
among youth and with staff 



Mean 



Alpha 



4.21 out of 7 



0.77 



Youth show positive affect to staff 

Youth are friendly and relaxed with one another 

Youth respect one another 

Staff show positive affect toward youth 

Staff engage personally with youth 

Staff provide guidance for positive peer interactions 

Staff use positive behavior management techniques 

Staff are equitable and inclusive 



Task-Oriented 



Mean 



Alpha 



The activity is organized with clear goals, and youth and 
staff are engaged and attentive. 



5.07 out of 7 



0.81 



Activity is well organized 
Youth are on task 

Staff communicate goals, purposes, and expectations 
Youth listen actively and attentively to peers and staff 
Staff attentively listen to and/or observe youth 



Exhibit reads: The skill-building observational scale had a mean of 3.60 points out of a possible 7 points and an alpha 
of 0.88. 
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Analyses of these observational data revealed that OST activities are generally successful 
at creating warm, welcoming, yet structured environments. However, programs were less 
successful in their efforts to implement active learning opportunities through activities that built 
on each other in a sequenced manner. On average, observed activities were rated highest on the 
task-oriented scale, which measured the extent to which the activity goals were clear and specific 
(5.07, out of 7). Activities also rated high on the relationship-focused scale, which measured the 
extent to which activities developed personal and social skills (4.21, out of 7). In contrast, 
evaluators observed less evidence of activity quality based on the extent to which the activity 
built on skills and content previously learned (3.60, out of 7, on the skill-building scale) or 
engaged youth in active, hands-on learning experiences (1.75, out of 7, on the active learning 
scale). 



However, evaluators found notable differences in ratings on the skill-building and active 
learning observational scales based on the content of the activity observed, suggesting that some 
activity types lend themselves more easily to implementation that reflects those quality 
components. In particular, art activities averaged 4.30 points, out of 7, on the skill-building 
scale, compared to 3.30 points for all other activities ( r ES = .35) 5 , indicating that art activities 
were more likely to involve a progression or practice of skills and to challenge youth 
intellectually and/or creatively. Not surprisingly, activities categorized as “open and 
unstructured” scored lower on the skill-building scale than other activities (2.28 points, compared 
to 3.82; r ES = .39). 



In addition, academic enrichment activities scored notably higher than non-academic 
enrichment activities on the active learning scale (2.14, compared to 1.64; r E s = -22). These 
results suggest that academic activities were more likely to provide youth with opportunities to 
engage with their learning and apply their skills than were other activities. In contrast, 
homework activities scored lower on the active learning scale than did other activities (1.46, 
compared to 1.86; r ES = .26). 

Programs in the site visit sample took varied approaches to implementing activities that 
met quality criteria. One program blended academic support with arts-based enrichment 
activities for the purpose of increasing youth engagement and learning. These activities also 
created active, hands-on learning opportunities for youth. A youth participant described this 
active learning process in a drama activity: 

I like the drama aspect of the after-school program. I learn a lot from it. We don ’t just 
learn how to act. We learn the stuff we ’re going to act about and the history behind it. 
We did a thing a couple of months ago with Dominican Independence Day. We learned 
about their whole history and then did a play and dancing. It’s really in-depth and I like 
that. 

Some programs also offered active learning through youth leadership opportunities. For 
example, a middle-grades program sponsored a youth council that engaged participants in 
activities to improve their communities. One group of participants in this program decided to 
begin a project on global wanning. They conducted research and planned a recycling campaign. 



5 For these analyses, r ES refers to the effect size measure for non-parametric statistical tests. 
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As one youth member said, “We go to conferences and go to other schools to speak about our 
school. We speak for the youth in our school. ... A lot of people don’t have a voice, so we speak 
for them.” The program director explained, “These are kids who have been empowered to get 
results for something they want to do.. . . This was a youth-led process. I assigned an adult to 
supervise the process but not get involved in the process.” 

Programs were often explicit in their approach to fostering positive relationships within 
the program environment. For example, one program organized its activities around a set of core 
assets, each representing a different social value such as sharing, giving compliments, and 
making safe choices. The program focused on one asset at a time, and the asset permeated all 
youth activities. Each day the program began with an opening assembly that set the tone for the 
rest of the day. In this session, the program director led a brief activity emphasizing the current 
asset. Staff members then worked to bring the featured asset into their lesson plans so that 
different activities built on one another. According to the program director: 

Our biggest strength is daily assembly. It is wonderful because it gives students 
transition time between school and after-school. I set standards as the program leader in 
the assembly. . . I have gotten so much support and recognition from day teachers who 
can hear what goes on in assembly. We are very mindful of language and tone [that we] 
use with students, and then we see the kids model our behavior. 

Another program implemented clear structures in which youth can express their opinions 
and air their frustrations and concerns. According to the program director: 

We have a community circle to voice ideas. They’re learning, it’s a process. Kids come 
with a lot of baggage and problems. We have a social worker come here during the day 
and after school. Our motto is “Be a buddy, not a bully. ” To reduce violence, [we] give 
kids alternate ways to be heard. Kids are free to come and fill out a form and talk with a 
group leader. If the group leader is still having a problem, the social worker comes and 
picks up the kid. 

Similarly, a middle-grades program held rap sessions on Fridays, when youth and staff 
talked about issues important to the youth participants. The program director noted that these 
sessions were beneficial for the program because they promoted closer, more open relationships 
between staff and participants: 

I would recommend sitting down and getting kids in a comfort zone to talk about what’s 
on their mind. It helps throughout the week. It helps you understand them a little more. 
You are not as quick to discipline because you know where they ’re coming from. It helps 
with the relationships with me and the staff. It helps my relationship with them. I share 
in those sessions, too. 1 share personal things as well. It helps. 
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Positive Relationships 



Research on the quality of out-of-school time programs suggests that staff can enhance 
youth engagement and learning by creating program structures that encourage youth to feel 
respected and supported by their peers and by staff members (Grossman et ah, 2007). To assess 
the extent to which OST programs contributed to the development of these positive relationships, 
evaluators analyzed participant reports of their interactions in the program. 



Positive Interactions with Peers 

Evaluators created a scale based on youth survey questions about the quality of youth 
interactions within the OST program (displayed in Exhibit 10), in order to measure the extent to 
which programs provided opportunities for youth to develop positive relationships with their 
peers. Overall, youth were overwhelmingly positive in their assessment of their social 
interactions within the program (average scale score of 3.32, out of 4 on the survey scale), 
although middle-grades and high school youth responded somewhat more positively than 
elementary-grades participants (3.35 and 3.37, compared to 3.28; d=. 11 and d=. 1 5, respectively). 
These high scores are an indication that OST programs were successful in creating welcoming 
social environments for youth to connect with each other and develop friendships — in their 
responses to individual items in the survey scale, 64 percent of youth agreed a lot that they have 
a lot of friends in the program, and an additional 26 percent agreed a little. 

Exhibit 10 

Participant Reports of Interactions with Peers, in Percents (n=6, 127) 



In this program I ... 

Have a lot of friends 
Have a good time playing with other students 
Get to know other students really well 
Get along with other students 
Like other students 
Can really trust the other students 

0 20 40 60 80 100 

■ Agree a lot □ Agree a little 

Exhibit reads: Sixty-four percent of participants surveyed agreed a lot and 26 percent agreed a little that they have a 
lot of friends in their program. 
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Positive Interactions with Staff 



Another measure of the quality of the environment of an out-of-school time program 
involves assessment of interactions between youth and staff. Evaluators created a scale 
measuring youth respondents’ assessments of their interactions with and relationships with staff 
members, based on the survey items summarized in Exhibit 1 1 . As with the peer interactions 
scale, OST participants responded very positively to this scale (average scale sore of 3.35, out of 
4), with high school youth responding somewhat more positively than either elementary-grades 
or middle-grades youth (3.46, compared to 3.3 1 for both elementary- and middle-grades youth; 
d =. 21 and d=. 20, respectively). 



Effective Program Partnerships and Supports 

The second year of the OST evaluation found positive associations between certain 
program staffing structures and youth outcomes. In particular, evaluators found positive 
correlations between diversified staffing patterns (including teen staff) and youth reports of 
academic benefits and motivation (Russell et ah, 2008). In a related finding, a recent study of 
the Philadelphia Beacons found positive relationships between staff skills and youth ratings of 
program quality, including staff skills in managing group dynamics to foster respect and support 
youth learning (Grossman et ah, 2007). 



Exhibit 11 

Participant Reports of Interactions with Staff, in Percents (n=6, 1 12) 



In this program ... 

Staff treat me with respect 

Staff think I can learn new things 

Staff help me try new things 

Staff think I can do things well 

Staff always try to be fair 

Staff really care about me 
I can talk to staff about things that are bothering 
me 

Staff care what I think 
Staff always keep their promises 

0 20 40 60 80 100 

■ Agree a lot □ Agree a little 

Exhibit reads: Sixty-four percent of participants surveyed agreed a lot and 26 percent agreed a little that staff treat 
them with respect. 
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Staffing Patterns 



Findings from previous years of the OST evaluation suggested that employing a diverse 
mix of staff and a master teacher contributed to program quality. In Year 3, OST programs 
continued to employ staff members with diverse backgrounds, as illustrated in Exhibit 12. The 
majority of programs (82 percent) hired college students, and 69 percent of programs hired at least 
one professional specialist (e.g., a professional artist, coach, or dancer). In addition, 63 percent of 
programs had at least one certified teacher on staff, and 61 percent hired teen staff members. 

The number of paid staff varied considerably across OST programs. Program directors 
reported between one and 65 staff members on payroll, with a median of 15 paid staff. More 
than half of programs (52 percent) enhanced their staff capacity by recruiting a median of three 
volunteer staff members. 



Not surprisingly, the median hourly wage varied considerably by staff member 
characteristics and background, with certified teachers receiving $30 on average, specialists $20, 
college students $11, and teen staff $8 per hour. Other adult staff were paid between $ 12 and 
$16 per hour. With funding a primary concern — 74 percent of directors reported that not being 
able to afford the competitive salaries necessary to hire qualified staff was a challenge — OST 
program directors maintained a balance of certified and professional staff members and younger, 
less experienced staff members. Accordingly, certified teachers and specialists were hired for a 
median of nine hours per week, compared to a median of 15 hours per week for all other staff 
types. Fifty-one percent of OST program directors reported having a staff member assigned to 
be a master teacher or educational coordinator within the program. 



Exhibit 12 



Percent of Programs Hiring Staff with Varied Characteristics and Backgrounds 

(n=456) 



Types of staff hired: 

College students 
Specialist staff 
Adults with college degrees 
Certified teachers 
Teen staff 

Adults without college 
degrees 

School day professionals 
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Exhibit reads: Eighty-two percent of program directors reported hiring college students to work in their programs. 



17 



Exhibit 13 indicates that OST programs were deliberate in the roles they assigned to 
certified teachers and specialists within the programs. Certified teachers were employed mainly 
to provide academic support to programs by leading academic activities (72 percent of programs) 
and tutoring (7 1 percent), while specialists were hired primarily for non-academic enrichment 
activities such as arts and sports (88 percent). College students, in contrast, played roles across 
program activities, including tutoring youth (88 percent) and assisting with enrichment and 
academic activities (82 percent and 78 percent, respectively). More than two-thirds of programs 
also employed college students to lead enrichment and academic activities (73 percent and 70 
percent, respectively). Teen staff were mostly hired as tutors (80 percent) and activity assistants 
(82 percent). 



Exhibit 13 

Distribution of Staff Roles by Staffing Category, in Percents 



Staff Categories 


Tutor 

Youth 


Lead 

Academic 

Activities 


Lead Other 
Types of 
Activities 
(Arts, 
Sports) 


Assist with 
Academic 
Activities 


Assist with 
Other 
Types of 
Activities 


Train Other 
Staff 
Members 


College students 


88 


70 


73 


78 


82 


32 


Teen staff 


80 


42 


49 


76 


82 


6 


Certified teachers 


71 


72 


47 


60 


48 




Specialists 


30 


28 


88 


33 


56 


41 



Exhibit reads: In 88 percent of OST programs, college students had responsibilities for tutoring youth. 



Consistent with these patterns, programs visited in the evaluation typically relied on 
young staff, including college students, to form the core of their OST staff. These programs 
often hired students who were pursuing studies in education and had experience working with 
youth or participating themselves in an OST program. Directors described the benefits of hiring 
these younger staff members: 

We always try to get former participants. They know each other from the neighborhood. 
It helps with retention as well, of the staff and of participants. It’s someone they 
[participants] can talk to, someone they can relate to. 

[Younger instructors] can relate to the students better. I feel that the children are able to 
relate more to them. They listen more. They think, “This person isn ’t that far from 
where I am. ” It’s that bonding, the personality. 

High school students are the ones who are learning how to run these programs. We 
empower them to do a little bit of everything. It’s the same thing with the [participants] . 
It comes all the way down. They get to see these high school and college students as role 
models doing great jobs. 
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Youth also appreciated having younger staff members work with them, because they felt 
better able to forge personal connections: 

They are like teenagers. They are not teenagers, but they can talk with us more closely 
than a grown-up coulcl. It is like ... imagine that a teenager came in to teach a class — 
they would be able to talk to us more easily than a teacher. I like teachers. They are nice 
and you learn new things from them, but the young staff are different; you can talk to 
them more easily. 

The staff are modernized. They are into what we are into. They aren ’t old-school. 

It’s good for the teachers to be closer to our age, because they can understand where 
we 're coming from. They ’re close to our age. If we have problems, they can give us 
advice. One actucdly lives in my building, and 1 see her on a regular basis. We have a 
different relationship on the block than we do in the program. 

To complement and support younger staff members, some programs also hired certified 
teachers or other experienced staff who could provide ongoing mentoring and supervision. One 
director described this as a “checks and balances system” for staff structure and noted that 
“having the younger staff paired with the older staff is great because they can learn from each 
other and create a nice sort of interaction. The older staff provide a maturity that the younger 
staff cannot bring.” A common approach was to employ certified staff during academic and 
homework help activities, both to provide targeted and high-quality help to students and to 
support the development of teaching skills in the younger staff: “The tutors really have gotten a 
chance to see how the teachers interact with the students and see how they interact in a 
classroom setting.” 

However, hiring certified teachers tended to increase the staffing expenses for programs 
and required program leaders to plan their program budgets to allow for the higher wages 
typically earned by teachers. These choices necessitated trade-offs in other program areas: 

“You have to really make use of your budget — you have to allocate enough money to hire the 
qualified staff. A lot of [organizations] maybe don’t budget enough, or say they’ll just hire 
students, or take out more for administrative expenses. [But] if you have a lot of [staff] turnover 
that can destroy a program.” In one program that had not budgeted to hire any certified teachers, 
the program director admitted that the quality of the academic component of the program had 
suffered: 

If we could afford certified teachers, that would be great. They come at their own rate, 
they are expensive... . It would help during the homework and academic portion. They 
would know how to address kids on that level. I have good staff who know how to deal 
with the kids, but not necessarily [how to help kids with homework] . It is easier for staff 
to motivate students during activity time, when the activity is their expertise. 
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Building Staff Capacity 

With the launch of the OST initiative in 2005, DYCD contracted with the Partnership for 
After-School Education (PASE) to provide technical assistance and professional development 
workshops for OST program staff. In survey responses, 88 percent of program directors reported 
that PASE was a primary source of technical assistance and training for themselves and their 
staff members. Other primary sources of technical assistance included the program’s provider 
organization (67 percent) and its DYCD program manager (40 percent). 

Various professional development opportunities and tools were available to OST 
programs through the initiative, as illustrated in Exhibit 14. Program directors most frequently 
reported that their staff received professional development through staff meetings at the program 
(86 percent), internal staff orientations (66 percent), and off-site workshops (62 percent). In 
addition, 27 percent of program directors reported receiving on-site consultations from PASE or 
other TA providers in Year 3 of the initiative. This represents a decrease from the 64 percent of 
directors who reported on-site consultations in Year 2. The change is consistent with DYCD’s 
focus on providing more in-depth assistance to fewer programs in the third year. 



Exhibit 14 

Percent of Programs Reporting Use of 
Various Professional Development Opportunities (n=456) 



Technical assistance tools or professional development opportunities that staff have used or 

participated in: 



Staff meetings at the program 
Internal staff orientation 
Off-site workshops 
Staff meetings at the provider organization 
On-site consultations by PASE or other TA providers 
Coaching/mentoring opportunities 
NYSAN Quality Self-Assessment tool 
Institutes or conferences 
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Exhibit reads: Eighty-six percent of program directors reported holding staff meetings at the program. 
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Programs in the in-depth sample relied heavily on internal professional development, 
which included review of lesson plans and regular staff meetings. More than half of program 
directors (54 percent) reported that they required most or all staff to submit activity plans; only 
14 percent never asked any staff to submit activity plans. In addition, 42 percent of directors 
held staff meetings at least once a week; another 3 1 percent held meetings two or three times a 
month. Program directors said that these meetings were a good opportunity for staff to discuss 
program improvements and share ideas: 

It’s also time for [staff] to unwind. They can say, “Oh, this child is doing that... ” Then 
they talk to each other and give advice. They help each other in that way. It helps that 
they have a good rapport with each other. 

In interviews, some directors also described conducting more formal evaluations with 
their staff members. In one program, for instance, the director met with staff to conduct formal 
evaluations two to three times per year, and staff were required to submit quarterly summaries of 
their work, assessing their progress and delineating any challenges they are facing. During 
evaluations, the director and staff members discussed not only how the staff member was 
performing but also how he or she could sharpen or improve his or her work. The director gave 
staff specific feedback on how to incorporate content and/or improved delivery strategies into his 
or her work with youth. The director explained that she often helped a staff member plan a 
lesson as part of this evaluation meeting. After the staff member had a chance to implement the 
lesson, they met again to discuss what worked and what didn’t work. The director said that she 
believed that supporting staff professionally was very important to helping them improve their 
work with youth. Directors and staff commented: 

[You must] treat the staff well for a successful program. Be firm and fair. Most staff will 
agree with the evaluation [I give them]. I cdways give people chances to grow with lots 
of feedback and guidance. It’s not the right place if you can 7 get it after a period of time 
with structured help. 

We do a lot of debriefing after professional development. Next week’s session will 
involve staff implementing a skill they learned. We do a lot of check-in where we talk 
about challenges and accomplishments. We provide advice or suggestions or 
recommendations. 

We shared different experiences that we have had, and we learn about different 
activities — then you can take them back to the site. Honestly, when other staff members 
present what they learn at their trainings, I sometimes feel like I was there myself. It is 
really good. 

Data from DYCD’s records of technical assistance offered in partnership with PASE 
confirmed that the focus of the OST initiative’s professional development shifted over time to 
provision of on-site, in-depth technical assistance to particular programs to meet their specific 
needs. In both the first and second years of the initiative, 93 OST programs completed at least 
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0.5 full-day equivalent (FDE) of technical assistance, with a median of 1.5 FDEs. 6 Programs 
that received technical assistance in Year 1 were most frequently assisted in developing 
programs/activities (43 percent), using DYCD Online (36 percent), and behavior/classroom 
management (25 percent). In Year 2, technical assistance was most frequently directed toward 
developing programs/activities (50 percent), behavior/classroom management (40 percent), and 
using DYCD Online (25 percent). In Year 3 of the OST initiative, fewer OST programs 
participated in technical assistance, but those who did participate received more intensive 
assitance. Fifty-four OST programs completed a minimum of 0.5 FDEs of technical assistance 
in Year 3, with a median of 3.5 FDEs. The most frequent technical assistance topics addressed 
in Year 3 were behavior/classroom management (52 percent), developing programs and activities 
(35 percent), and program administration, management, and organization (19 percent). This shift 
in topics may reflect DYCD’s increased focus on organizational development as well as the 
increased stability of OST programs, leading to greater focus on improving program capacity and 
quality. 



Program directors appreciated the hands-on nature of site-based trainings. One noted that 
“what’s really wonderful about the site -based training ... is that you select a provider and a topic 
area, but then you can kind of custom-fit the workshop to the needs of the staff in the program.” 
Another anticipated a site-based training on youth development and behavior management, and 
explained that “[PASE] had a consultant come and meet with us before, and she gave 
suggestions and resources. I know this workshop will be really good for the staff.” This director 
explained the benefit of having an outside perspective for staff training: “It resonates more when 
it comes from someone other than your supervisor. They might have new ideas I didn’t talk 
about.” 



Partnerships with Schools 

A statewide study of after-school programs in Massachusetts found that after-school 
programs that nurtured strong relationships with teachers and principals also improved 
participants’ homework effort and completion, behavior, and initiative (Miller, 2005). In the 
third year of the OST initiative, program directors reported regular communication with school 
staff in several areas. As shown in Exhibit 15, more than half of program directors reported 
communicating with school administrators or staff at least monthly about: the needs or progress 
of individual students (61 percent), issues related to sharing classroom space (56 percent), 
homework assignments (56 percent), and student discipline policies (50 percent). On surveys, 
directors reported high levels of satisfaction with their communication with school staff. Sixty- 
one percent of program directors said that receiving responses to requests to coordinate services 
or resources with school staff was not a challenge; only 6 percent reported that this was a major 
challenge. 



6 OST programs may self-refer to technical assistance or be referred through their DYCD program manager. A self- 
referral generally indicates that a program is committed to the technical assistance process and is actively seeking 
out consultation. 
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Exhibit 15 

Communication with Schools, in Percents (n=456) 




Exhibit reads: Sixty-one percent of program directors reported discussing the needs or progress of individual students 
with school personnel at least once a month. 



In interviews, OST staff reported ongoing, informal communication with participants’ 
school-day teachers: 

We have a lot of meetings, and we have access to the principal. Even though he assigns 
assistant principals to oversee the program, we have access to the principal to tell him 
what we think is working and what is not. A good relationship with the principal is the 
backbone of success for an after-school program. 

I’ll ask, “How is so-and-so doing in class?” Their teacher will say, “Yes, they did this 
homework, they ’re working independently. ” 

We’ve developed such a great relationship with the teachers that I can go into the 
classroom during the day and ask about a particular student and whether he has 
homework, and the teacher will tell me. 



Partnerships with Parents 

Programs reach out to the families of participants to engage them and to meet youth 
needs more effectively. As shown in Exhibit 16, programs use varied methods to communicate 
with families. Nearly all (91 percent) have conversations with parents over the phone at least a 
few times a month; 83 percent meet in person with parents that frequently. In interviews, 
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program directors described the importance of being accessible to parents. According to one 
elementary-grades program director: 

I like to walk around, specifically at dismissal time. I stand out there as the parents come 
in. 1 make myself accessible. Parents have things they want to ask, but if they don 7 see 
me, they ’ll forget about it. My strategy is to be accessible. ... That way parents know that 
I’m here, their children are safe, I know what’s going on. 

Recognizing the importance of family connections, programs also rely on family or 
parent liaisons to engage families and encourage high rates of participation. Forty-five percent 
of programs employed someone as a parent or family liaison. In one program, for example, the 
family liaison was a paid part-time position described as “the first point of contact with 
families.” The liaison was responsible for contacting families when a child was sick or having 
problems during the program and also for monitoring youth participation, including data entry 
for DYCD Online. 



Exhibit 16 

Communication with Parents, in Percents (n=456) 



How often do you... 



Have conversations with parents over the phone 



Meet with one or more parents 



Send materials about the program home to parents 



Hold events or meetings to which parents are invited 

Hold events or meetings to which community members are 
invited 

0 20 40 60 80 100 

Percent responded at least a few times a month 



Exhibit reads: Ninety-one percent of program directors reported having conversations with parents over the phone at 
least a few times a month. 
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4. Evidence of Youth Outcomes 



This section reviews evidence of youth outcomes in three areas, including youth 
engagement in OST programming, social development of participants, and their educational 
development. While program engagement by itself does not constitute evidence of positive 
youth development, it is directly instrumental to the emergence of growth in the other two 
outcome domains considered in this chapter. 



Program Engagement 

Recent research identifies the following four important measures of out-of-school time 
program engagement (Chaput, Little, & Weiss, 2004), each of which were used in this evaluation 
to assess the extent of youth engagement in the OST initiative: 

■ Enrollment: Whether youth spend any time in an OST program 

■ Breadth: The variety of OST activities in which youth participate 

■ Intensity: The amount of time that youth spend in a program during a given 
period 

■ Duration: A youth’s attendance history across program years 



Enrollment 

The first steps in achieving high levels of program enrollment are effective outreach and 
recruitment. Throughout the OST initiative, program directors reported consistent use of 
recruitment strategies to meet their enrollment and participation goals. In Year 3, the majority of 
Option I program directors (90 percent) reported that they offered open enrollment to all youth 
who were interested in attending the program. In addition, more than half of the program 
directors reported that they targeted youth who were recommended by school-day teachers or 
counselors (58 percent) and youth with siblings already attending the program (55 percent). 

Over three-quarters (76 percent) of elementary-grades programs reported that they had a 
waiting list, as did 25 percent of middle-grades programs and 19 percent of high school programs. 
Programs that had a waiting list were significantly more likely than were programs that did not 
have a waiting list to target recruitment efforts to youth who were eligible for subsidized child care 
(e.g., former ACS) (64 percent, compared to 42 percent; V=22), and to youth with siblings already 
in the program (61 percent, compared to 41 percent; V=.20). Not surprisingly, programs with a 
waiting list were less likely to have open enrollment for all interested youth than were programs 
without a waiting list (50 percent, compared to 74 percent; V=.15). 
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Analyses of data from DYCD Online indicate that in the third year of the OST initiative, 
Option I programs on average exceeded their targeted enrollment levels. Option I programs had 
a target enrollment overall of approximately 63,000 youth, based on the contracts awarded by 
DYCD. As shown in Exhibit 17, these programs actually served a total of about 64,500 students 
from September 2007 through June 2008. 7 



Exhibit 17 

Targeted Enrollment and Actual Number of Students Served, 

by Grade Level 



Grade Level 


Targeted Enrollment 


Students Served 


Elementary 


37,551 


37,830 


Middle 


13,499 


14,860 


High 


12,146 


11,834 


Total 


63,196 


64,524 



Exhibit reads: The enrollment target for elementary programs was 37,551 youth, 
and elementary programs actually served 37,830 youth. 



However, some individual programs struggled with meeting their targeted enrollment, as 
measured by the number of slots available for participants based on the program’s contract with 
DYCD. As shown in Exhibit 18, in the third year of the OST initiative, 69 percent of elementary 
programs met or exceeded their enrollment targets, as did 70 percent of middle-grades programs 
and 59 percent of high school programs. 



7 DYCD permitted programs to over-enroll participants by a certain percentage. Option 1 elementary-grades 
programs were permitted to over-enroll by 20 percent, middle-grades-programs by 25 percent, and high school 
programs by 30 percent. In its program-level monitoring efforts, DYCD calculated participation rates using the 
budgeted, rather than actual, enrollment count. For example, an elementary-grades program that was budgeted to 
serve 100 participants and enrolled 120 participants could achieve its 80 percent participation target through an 
average daily attendance of 80 students. 
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Exhibit 18 

Percent of Programs Meeting Enrollment Targets, By Grade Level 
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Exhibit reads: Thirty-one percent of elementary programs were under-enrolled, while 69 percent of elementary 
programs met or exceeded their enrollment targets. 



Breadth 

As described earlier in the report, evaluators collapsed the DYCD Online activity 
categories into academic enhancement, arts and culture, community building, life skills, career 
and work, and recreation activities. Analyses yielded an index depicting the different types of 
activities in which each youth participated, ranging from 1 to 6, which indicated the breadth of 
activities in which OST programs engaged participants. Participants in elementary-grades 
programs engaged in the widest range of activities, with an average of 3.4 (out of 6) activity 
types, while middle-grades students participated in an average of 3.0 activities. As expected, 
high school participants attended a more focused set of activities: high school youth participated 
in an average of 2.1 different types of activities in Year 3, with 53 percent participating in 
academic enhancement activities and 44 percent in recreation activities. These were also the 
most commonly attended activities for elementary-grades and middle-grades participants, with a 
substantially higher rate of participation for both activities among these younger students: 96 
percent of elementary-grades participants and 9 1 percent of middle-grades participants attended 
academic enhancement activities, and 75 percent and 83 percent attended recreation activities, 
respectively. In addition, more than three-quarters of elementary-grades students participated in 
arts and culture activities (78 percent). These numbers reflect the patterns seen in earlier years of 
the initiative. 
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Intensity 

Based on the program-level OST participation goals established by DYCD, evaluators 
calculated the minimum number of hours each Option I participant was expected to receive 
during Year 3, as shown in Exhibit 19. 8 As in Years 1 and 2, programs reached a high standard 
of participation, with elementary-grades programs continuing a trend of improving their average 
rate of participation. On average, elementary-grades participants attended 377 hours during the 
year, compared to the 432 hours they were expected to attend. This represents an average of 87 
percent of targeted hours, an increase over both Year 1 (72 percent) and Year 2 (83 percent) of 
the initiative. In elementary-grades Round 2 programs, which started operations in winter 2008, 
the evaluation computed a pro-rated 173 hours of expected participation, and, on average, 
participants in these programs surpassed that expectation, achieving 180 hours of participation 
on average. 

Older youth also surpassed their targeted number of OST participation hours, on average. 
Middle-grades participants as a group achieved their targeted number of hours of participation: 
on average, middle-grades participants attended 218 hours of the 216 hours expected at the 
middle-grades level (101 percent of the targeted hours). This was about the same as the 213 
hours that middle-grades participants attended on average in Year 2, which was a substantial 
increase over the 159 hours attended, on average, by middle-grades participants in Year 1. 
Finally, high school participants exceeded their targeted number of hours of participation, 
attending on average 92 hours in the second year of the initiative, 16 hours above their target of 
76 hours, although somewhat fewer hours than in the previous two years of the initiative (97 
hours in Year 1 and 105 hours in Year 2). 



Exhibit 19 

OST Participants’ Actual and Targeted Mean Attendance, in Hours 



Hours of Attendance 


Elementary 
(Round 1 only) 

n=25,812 


Middle 

n=14,860 


High 

n=1 1,834 


Targeted hours 


432 


216 


76 


Actual hours (mean) 


377 


218 


92 


Actual (mean) as 
percent of target 


87% 


101% 


121% 



Exhibit reads: The targeted level of participation at the elementary level is 432 hours, 
and elementary participants actually attended for an average of 377 hours, 
or 87 percent of targeted hours. 



At the elementary level, programs are expected to offer programming for a minimum of three hours a day, five 
days a week, for 36 weeks, plus 10 hours a day over 20 vacation days for a total of 740 hours. For purposes of 
computing the expected number of hours of participation, evaluators used the daily service-availability guidelines 
but excluded the OST service hours expected on school-closing days, which produced a total of 540 hours. 

Adapting DYCD’s program-level participation-rate requirement, evaluators set the expected number of hours for an 
elementary-grades participant at 80 percent of this level, or 432 hours. Using similar calculations, the expected 
numbers of program hours for middle-grades and high school participants were set at 216 and 76 hours, respectively. 
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Overall, 42 percent of elementary-grades participants attended for at least the targeted 
number of hours. Somewhat higher percentages of middle and high school participants met the 
attendance targets: 47 percent of middle-grades participants did so, as did 49 percent of high 
school participants. As shown in Exhibits 20-22, some youth at each grade level far exceeded 
their targeted number of hours, and the attendance rates of those who fell below the targeted 
range varied to a similar extent. 



Exhibit 20 

Percent of Participants Who Met Participation Target, 
Elementary-Grades Youth 



100 



C 

(0 

Q. 

■| 60 



0 

1 4 0 

o 

1— 

<b 

Q_ 

20 



0 



Exhibit reads: Fifty-eight percent of elementary-grades participants did not attend their OST programs for the 
targeted number of hours, while 42 percent met or exceeded the target. 







Below Targeted Hours: 58% 


Met or Exceeded 
Targeted Hours: 42% 


♦- ♦ ♦ ♦ ♦ ♦ 





T 1 1 1 1 1 1 1 r— 1 1 1 1 r 



0 50 100 150 200 250 300 350 400 450 500 550 600 650+ 

Participant Hours Attended 

— ♦ — Elementary grades participants from Round 1 programs (n=25,812) 



Evaluators also examined the relationship between whether a program met its targeted 
enrollment goal and whether participants in the program met their targeted number of hours, and 
found a significant positive relationship at all grade levels, indicating a greater proportion of 
participants achieving the participation goal in programs that also successfully enrolled large 
numbers of students. Conversely, programs that were likely to struggle to achieve their enrollment 
goal also struggled to have individual participants attend regularly. Among elementary-grades 
programs that met their target enrollment goal, 35 percent of participants were at or above their 
targeted hours, compared to 14 percent of participants in elementary-grades programs that did not 
meet their goal. The pattern was similar for middle-grades OST programs (50 percent compared to 
40 percent) and for high school programs (53 percent compared to 39 percent). 
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Exhibit 21 

Percent of Participants Who Met Participation Target, 
Middle-Grades Youth 
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Exhibit reads: Fifty-three percent of middle-grades participants did not attend their OST programs for the 
targeted number of hours, while 47 percent met or exceeded the target. 



Exhibit 22 

Percent of Participants Who Met Participation Target, 
High School Youth 
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Exhibit reads: Fifty-one percent of high school participants did not attend their OST programs for the targeted 
number of hours, while 49 percent met or exceeded the target. 
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Duration 



As shown in Exhibit 23, 6,283 youth participated in all three years of OST school-year 
programming, in the programs that had begun in September 2005 (Round 1 programs). Almost 
22,000 youth participated in two years of OST programming, either in Years 1 and 2 or in Years 
2 and 3, while 90,826 youth participated in just one year of programming (Year 1, Year 2 or 
Year 3). Approximately 22,000 youth only participated in summer programming. An additional 
12,286 youth participated for part of the third year of the initiative, in the elementary-grades 
programs that were first funded in January 2008 (Round 2 programs). 

Exhibit 23 

Years of OST Participation, Round 1 Programs 



Duration of Participation 


Number of 
Participants 


One year of participation only 


90,826 


Two years of participation only 


21,501 


Three years of participation 


6,283 


Only participated in summer(s) 


22,178 


Total unique participants 


140,788 



Exhibit reads: 90,826 youth participated for a single year of OST 
programming in the Round 1 programs that began in September 2005. 

Evaluators examined the rate at which Option I OST participants from the 2006-07 
school year (Year 2) re-enrolled in the same OST program in the 2007-08 school year (Year 3). 
Of the 43,503 participants who attended an OST program in the initiative’s second year and were 
eligible to return to that program, 16,918 (39 percent) enrolled in the same OST program in 
2007-08. This 39 percent retention rate is similar to the 37 percent retention rate in Year 2. 

These data underestimate the percent of Year 2 participants who desired to continue their OST 
participation for another year: OST programs typically enroll participants on a first-come, first- 
served basis each year, and returning participants are not guaranteed enrollment for a subsequent 
year. 



As shown in Exhibit 24, evaluators combined the measures of duration and intensity to 
compute the total number of hours each participant was exposed to OST programming after one 
year, two years, and three years of participation. Hours attended during the summer months 
were included in the totals for the year following that summer. On average, after one year of 
participation, OST participants of all ages in Round 1 programs had attended 227 hours of 
programming; after two years of participation they had been exposed to 658 hours of OST. 
Participants who were enrolled during each of the three years of OST initiative participated in an 
average of 1,168 hours over that time period. Among other things, these figures indicate that 
youth who participated for the longest duration had especially high levels of annual participation. 
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Exhibit 24 

Average Cumulative Hours of OST Participation, Round 1 



Duration of Participation 


n 


Hours of OST 
Exposure 


After one year of participation 


114,629 


227 


After two years of participation 


27,602 


658 


After three years of participation 


6,283 


1,168 



Exhibit reads: After one year of participation, youth had participated in an average 
of 227 hours of OST service. 



Social Development 

Recent studies of OST programs have found positive social development outcomes for 
youth participants. By creating positive environments for youth to interact with each other and 
with adults, and by offering activities that explicitly promote youth development, OST programs 
can contribute to youths’ sense of belonging within the program community and to the 
development of pro-social behaviors. An analysis of the program effects of the LA’s BEST 
after-school program, for example, found that participants were less likely than their peers to 
compile juvenile crime records in later years, suggesting that after-school participation 
contributed to positive social outcomes (Goldschmidt & Huang, 2007). 



Sense of Belonging 

The sense of belonging scale, created based on the youth survey items displayed in 
Exhibit 25, describes the extent to which participants felt connected to their OST program. 
Technical details about this and other youth outcome survey scales are included in Appendix C. 
Overall, participants reported a strong sense of belonging in the third year of the initiative, 
similar to their reports from previous years, with an average scale score of 3.38 out of 4. In 
particular, more than two-thirds of participants agreed a lot that they felt safe in the program (68 
percent) and 58 percent felt that the program was a “good place to hang out.” High school 
students were especially likely to report a strong sense of program connection (scale score of 
3.48, compared to 3.32 for elementary-grades participants and 3.38 for middle-grades 
participants; d=.25 and <r/=.16 respectively). 
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Exhibit 25 

Participant Reports of Sense of Belonging (n=6, 122) 



In this program I feel like ... 

I am safe 

This is a good place to 
hang out 

I belong 
I am successful 
I matter 
My ideas count 

0 20 40 60 80 100 

■ Agree a lot □ Agree a little 

Exhibit reads: Sixty-eight percent of participants agreed a lot that they are safe in their program. 

Participants interviewed confirmed this sense of belonging and reported in group 
interviews that they enjoyed spending time in their OST program: 

I like coming because I get to spend time with my friends and talk to my friends. 

I like [the program] because it is giving me a lot more friends and a lot more speciality 
[makes me feel special]. I feel that I am appreciated in this program, because everyone 
treats me very nice. 

When I came, I was nervous because I didn 7 know anyone, then 1 made friends. I get to 
meet new people. 

I feel I ’m wanted here. I have fun here. If I am wanted, 1 will keep coming. . . . The staff 
are friendly. You can talk to them and have a conversation. 




Pro-social Behavior 

To measure participants’ engagement in pro-social behaviors, evaluators asked middle- 
grades and high school participants to report on the frequency in which they engaged in the 
following positive behaviors in the month prior to the survey: 
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■ Helped someone stay out of a fight 

■ Told other students how I felt when they did something I liked 

■ Cooperated with others in completing a task 

■ Told other students how I felt when they upset me 

■ Protected someone from a bully 

■ Gave someone a compliment 

■ Helped other students solve a problem 

In the third year of the program, as in previous years, participants reported moderate 
levels of engagement in pro-social behaviors, with an average score of 2.50, out of 4, and no 
significant differences by grade level. Participants were most likely to report that they had done 
the following more than three times in the prior month: given someone a compliment (62 
percent), cooperated with others in completing a task (62 percent), and helped other students 
solve a problem (56 percent). 

In interviews, staff members described experiences in which youth developed strong, 
trusting bonds with program staff and with each other, developing their social skills: 

I had a kindergartener last year, and most of them are in the same class this year. They 
want to talk, they want to tell you everything. I have this one student — she would sulk 
and sit by herself all the time last year. This year, she gives me hugs, and she talks all 
the time. She is a lot more social, and she is a lot more confident. I think that it helped 
her. 

I had one child, she would not speak at all, she would sit there and start crying, she 
would not say anything. We would go to the bathroom and calm her down. She needed 
someone to just sit with and get that attention, and now she talks and brings her little 
sister. She wanted someone to pay her some attention. 

There were two girls who constantly fought, we would have to separate them, they would 
physically fight and verbally fight. And now they are pretty good, they even go to dance 
class together. I think that it is all because of family and seeing that everyone belongs, 
like a family we do not need to like everything, but because we are a family, we get along. 

Youth also described ways in which participation had helped them improve their social 

skills: 



1 have become more socicd with other people. Usually I don 7 recdly tcdk to other people, 
1 am very quiet. It helps me get cdong with everybody else and my personality grew 
more. It goes back to the staff— the staff and the kids that are attending make me feel 
happy. 

I wasn 7 recdly the [kind of] person that would be so friendly. But as I started to get into 
the program, [the director] helped us learn to tcdk to people in a way that they would 
want to be our friend again. 



34 




Educational Development 



Analyses examined the relationship between OST participation and the following 
measures of educational development: academic motivation, school attendance rate, academic 
benefits reported by youth, performance on the state English Language Arts (ELA) and 
mathematics tests, and credits accrued and Regent exams passed for high school students. 
Evaluation findings regarding educational development in other after-school and out-of-school 
time programs are mixed. 



Academic Motivation 

The academic motivation scale created for this evaluation measures participants’ reports 
of the extent to which they enjoy and engage in academic pursuits, which can be an important 
precondition to academic achievement. The scale employs the survey items displayed in Exhibit 
26. Participants reported an overall strong level of academic motivation in Year 3, as in prior 
years, with an average scale score of 3.34 out of 4. Most notably, 71 percent of participants 
agreed a lot that they try hard in school. Elementary-grades participants reported notably higher 
levels of academic motivation than did middle-grades or high school participants (scale score of 
3.34, compared to 3.25 for both groups of older youth; d =. 34 for both comparisons). 

Exhibit 26 

Participant Reports of Academic Motivation, in Percents (n=6, 1 18) 



Try hard in school 
Do well in school 
Try always to come to class prepared 
Pay attention in class 
Always finish my homework 
Enjoy math 
Enjoy writing 
Try to enjoy school 
Enjoy reading books for pleasure 

0 20 40 60 80 100 

■ Agree a lot □ Agree a little 

Exhibit reads: Seventy-one percent of participants agreed a lot that they try hard in school. 



In general I ... 
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School Attendance Rate 



Another measure of academic engagement is school-day attendance rate. DOE provided 
evaluators with daily school attendance data for OST participants who had attended the programs in 
the evaluation’s in-depth sample at any time during the initiative and for matched nonparticipants 
who attended schools that did not host a DYCD youth program and who had similar characteristics 
as the OST participants. Technical details about the matching process are described in Appendix E. 

Although closely matched on 
school and individual demographic 
characteristics, OST participants attended 
school significantly more frequently than 
did nonparticipants at baseline, prior to 
the youth’s enrollment in OST (92.6 
percent attendance rate at baseline for 
OST participants, compared to 91.3 
percent for nonparticipants, d= 0. 14). 

This suggests that higher attending youth 
self-select into OST programs. 

Participants continued to attend school at 
a higher rate than nonparticipants after 
one, two, and three years. After 
accounting for different baseline 
attendance levels, analyses found no 
notable differences in changes in school 
attendance rates over time between 
participants and matched nonparticipants. 

Academic Benefits 

In the third year of the OST 
evaluation, participants reported moderate 
levels of OST-related academic benefits 

on a scale developed based on youth responses to the survey items summarized in Exhibit 27. 

On average, participants had an average scale score of 3.06, out of 4, similar to the level found in 
previous years of the evaluation, with no notable differences by grade level. The most common 
academic benefit reported by participants was that the program helped them to finish their 
homework more often (54 percent agreed a lot). 

Interviews with youth participants revealed that the personalized instruction they received 
from staff in OST programs contributed to their learning: 

Tutors, they actually listen. Teachers [during the school day] are in a rush, they can ’t 

hear what you ’re saying. 



Program Structures to Support Academic 
Improvement 

A middle-grades program in the evaluation’s in-depth 
sample has a strong partnership with its host school 
and incorporates test preparation for the state ELA and 
math tests into the program schedule. Prior to the 
January test, participants study ELA three days a week 
with teachers from the day school. After the ELA test, 
students prepare for the math test in mid-March. 
Program tutors, who are college students, assist in 
classrooms during ELA and math prep periods, and 
then participants go to their chosen OST activities, 
including athletic and artistic enrichment activities. 

After ELA and math tests are completed for the year, 
participants use the first hour of the program to 
complete their homework. The program makes a 
concerted effort to keep participants engaged by 
approaching literacy and math activities with materials 
that differ from those used during the school day. 
According to the program director, having different 
instructional materials in the after-school program also 
helps teachers remain engaged. “What makes it easier 
for the teachers, they are not using the same program 
from the day school or in the Saturday program. It is a 
whole new program. It is something totally different. I 
didn’t want the kids to be burned out with doing the 
same thing in after-school.” 
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One time in math, they taught me fractions. I clidn 7 know how, so in the program I asked 
the tutors, [and they] taught me a lot. The next day at school I showed the teachers I 
could do it... My teacher asked, “Who was helping you? after school?” She scad that 
was good. 



Exhibit 27 

Participant Reports of Academic Benefits, in Percents (n=6, 1 18) 



This program has helped me ... 

Finish my homework more often 
Get better grades in school 
Solve math problems better 
Read and understand better 
Feel better about my schoolwork 
Write better 

Use computers to do schoolwork better 

0 20 40 60 80 100 

■ Agree a lot □ Agree a little 

Exhibit reads: Fifty-four percent of participants agreed a lot and 30 percent agreed a little that the program helped 
them finish their homework more often. 




Performance on ELA and math state tests. New York State administers annual tests in 
English Language Arts (ELA) and mathematics to students in grades 3-8 in January and March, 
respectively. Test-takers receive a scale score and one of four performance levels — ranging from 
Not Meeting Learning Standards (Level 1) to Meeting Learning Standards with Distinction 
(Level 4) — on the test. However, the scaling of student achievement test scores for the New 
York State tests poses special challenges for the analysis of test-score change. In particular, 
distribution of scale scores is neither identical across grade levels, nor does it follow a regular 
progression across grade levels. In addition, there is no standard for the expected gain between 
grade levels. This problem significantly complicates comparisons of scale scores from one grade 
to the next. 



As a solution, evaluators took the approach of standardizing the scale scores across 
grades, so that the range of possible test scores was 0 to 100 at each grade level and the mid- 
point of the possible scale scores for each grade level was always 50. The formula used to 
transfonn each student’s scale score at each grade level into a standardized score is: 
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Scale Score — Minimum Possible Scale Score 

Maximum Possible Scale Score — Minimum Possible Scale Score) 



x 100 



Using this method, differences in standardized scale scores across grade levels represent 
change in student perfonnance that are independent of the differences in the particular scale 
applied to the test scores at each grade level. Differences in the standardized scale scores across 
grade levels are thus expressed as differences in the proportion of possible scale-score points that 
a student earns in one grade level compared to the proportion earned at the next. 

Two important factors affect analyses of the relationship between OST participation and 
performance on the ELA and mathematics tests. First, because of the timing of test administration 
during the school year, it is unlikely that one-year participants would have received sufficient 
exposure to OST programming to make a significant impact on their test perfonnance in January 
and March. Second, the OST initiative operates within a context of significant educational reforms 
in New York City: beginning prior to the launch of the OST initiative in September 2005, New 
York City students have made steady achievement gains in both ELA and math, as illustrated in 
Exhibit 28. 



Exhibit 28 

Citywide Trends in Attendance and in ELA and Math Passing Rates 



Performance Indicator 


2002-03 


2003-04 


2004-05 


2005-06 


2006-07 


2007-08 


School Attendance (percent) 


89.5 


89.0 


88.6 


88.4 


89.2 


89.7 


ELA (percent at grade level) 


41.0 


41.1 


51.8 


50.7 


50.8 


57.6 


Math (percent at grade level) 


41.9 


46.7 


52.9 


57.0 


65.1 


74.3 



Exhibit reads: In 2002-03, the average school attendance rate was 89.5 percent. 
Source: http://schools.nyc.gov/Accountability/YearlyTesting/TestResults/default.htm 



For the analyses described here, OST participants and nonparticipants were closely 
matched on their baseline academic performance in both ELA and math, with no significant 
differences. The average ELA standardized score was 55.0 points for participants and 55.5 for 
nonparticipants; both groups had an average math score of 57.9 at baseline. Consistent with the 
city wide trends, both groups showed small improvements in performance over the course of the 
OST initiative, with no significant differences in the size of the gains between the two groups. 

High school performance. To assess the performance of high school OST participants 
compared to matched nonparticipants, evaluators examined the cumulative number of course 
credits accrued after each year of OST participation. Analyses found no significant differences 
between the groups on this measure. Analyses also examined the number of New York State 
Regents exams that participants and nonparticipants had taken and passed, in order to assess 
progress towards graduation, and found no significant differences. 
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5. Relationships Among Participation, Quality, and Outcomes 



The analyses summarized in the preceding chapters describe how OST programs 
performed on specific measures of program-implementation quality and also describe the overall 
patterns of participant outcomes. In addition, evaluators examined relationships among 
measures of youth participation, program quality, and youth outcomes in a series of 
complementary analyses, including correlational analyses (see Appendix F for technical details) 
and multi-level regression models (see Appendix G), presented below. 

In addition to the quality of programming, the duration and intensity of exposure to 
programming are important contributors to youth outcomes. For example, the evaluation of 
programs sponsored by The After-School Corporation (TASC) found that outcomes were 
greatest for students who participated in TASC programs on a regular basis for more than a year 
(Reisner, White, Russell, & Birmingham, 2004). These types of relationships and others are 
reviewed in this chapter. 



Relationships Between Participation and Youth Outcomes 

Evaluators analyzed the relationship between participation over the course of the 
initiative and youth social and educational outcomes for youth in the in-depth sample. These 
analyses focused on cumulative outcomes based on the most recent year of OST participation. 
For example, for a student who participated in OST for two consecutive years starting in 2005- 
06, the evaluation examined the relationship between total hours of OST during those two years 
and academic outcomes at the end of the 2006-07 school year. Details of these correlational 
analyses are shown in Appendix F. 

The number of hours of OST participation was positively correlated with school 
attendance rates after one, two, and three years of OST enrollment (r/=.17, r s =.13 and r s =.14, 
respectively). As noted earlier, on average OST participants attended school significantly more 
frequently than did nonparticipants at baseline, prior to the youth’s enrollment in OST (92.6 
percent attendance rate, compared to 91.3 percent, <7=0. 1 4), and continued to attend school at a 
higher rate than nonparticipants, with no notable differences between groups after accounting for 
baseline school attendance levels. However, as seen in Exhibit 29, OST participants who 
attended the program for fewer than 200 hours had lower school attendance than did 
nonparticipants (90.8 percent, compared to 91.2 percent), while participants who attended more 
hours of OST tended to have higher school day attendance rates. For example, students who 
attended from 201 to 400 hours of OST had an average school attendance rate of 92.5 percent, 
compared to 95.6 percent for students who attended OST for more than 1,000 hours. 

Analyses did not find other patterns of significant correlations between program 
participation and youth social or educational outcomes. 
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Exhibit 29 

Average School-Day Attendance Rate by Hours of OST Participation, All Grades 




Exhibit reads: Nonparticipants (attended 0 hours) had an average school-day attendance rate of 91.2 percent. 



Analyses found no notable relationships between levels of OST participation and 
performance on the New York State ELA and mathematics exams for youth in grades 3-8 or total 
credits earned by high school students. 



Relationship Between Program Quality and Youth Outcomes 

Evaluators also examined associations between measures of program quality and youth 
outcomes. Notable correlations between the evaluation’s measures of program quality and youth 
engagement, social, and educational outcomes are summarized in Exhibits 30-32 and presented 
below. These reports of correlations focus on those where there was a consistent pattern in the 
relationship between the measure of program quality and the youth outcome at two or more 
analysis time points. Because of the relatively small number of in-depth programs in the 
evaluation sample (15), these analyses could not be accurately sub-divided by the grade level 
served by the OST program. 
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Relationships Involving Rich Program Content 



Overall, the correlation analyses revealed positive relationships between the extent to 
which a program exposed youth to new experiences (as measured by aggregate youth reports) 
and several measures of youth social and educational outcomes (Exhibit 30). Fewer positive 
correlations were found between program-content range and measured youth outcomes. 

Exhibit 30 

Correlations between Features of Rich Program Content and Youth Outcomes 



Youth Outcomes 


Spearman Correlation Coefficient (r s ) 


Average Youth 
Ratings of Exposure 
to New Experiences 


Range of Program 
Content 


Program Participation 


After 1 year 


-.31 




After 2 years 


-.43 


.16 


After 3 years 


-.18 


.32 


Sense of Belonging 


After 1 year 


.24 


-.21 


After 2 years 


.24 


-.24 


After 3 years 


.26 




Prosocial Behavior 


After 1 year 






After 2 years 






After 3 years 


n/a 


n/a 


School Attendance 


After 1 year 






After 2 years 


.11 




After 3 years 


.22 




Academic Motivation 


After 1 year 




-.14 


After 2 years 


.15 


-.20 


After 3 years 






Academic Benefits 


After 1 year 


.17 


-.20 


After 2 years 


.23 


-.23 


After 3 years 






ELA Gains 


After 1 year 






After 2 years 






After 3 years 






Math Gains 


After 1 year 






After 2 years 






After 3 years 






Total Credits Earned (High School) 


After 1 year 




.25 


After 2 years 




.24 


After 3 years 


n/a 


n/a 



Note: Only correlations that are statistically significant and with effect sizes of at least .10 are included in this 
exhibit. Shaded cells indicate positive correlations. 



Exhibit reads: Total hours of participation after one year of OST enrollment and a program’s average youth 
reports of exposure to new experiences were negatively correlated (r s =-.31 ). 

■ Youth reports of their sense of belonging in the OST program were positively 
correlated with the program’s extent of exposing youth to new experiences after 
one, two, and three years of participation (r s =. 24, r s =.24 and r s =.26, 
respectively). 
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■ Youth engagement in the school day was positively associated with average 
program-level reports of exposure to new experiences, especially as measured by 
school attendance after two and three years of OST participation (r s =.11 and r s 
=.22, respectively). 

■ Youth reported higher levels of academic benefits after one and two years of OST 
participation when the OST program exposed youth to more new opportunities (r s 
= .17 and r s =.23, respectively). 

However, the extent to which a program exposed youth to new experiences was also 
negatively correlated with the total hours of youth participation in OST after one, two and three 
years (r s =-.31, r s =-.43, and r s =-.18, respectively). One possible explanation for this pattern is 
that youth who attend OST more become accustomed to the program offerings over time and 
therefore heighten their expectations for a continuing stream of novel experiences in the 
program. 

The breadth of OST program content in OST programs, measured by the number of 
different activity types offered by the program, was negatively correlated with several youth 
social and educational outcomes, including youth reports of their sense of belonging and of 
academic motivation and benefits. In contrast, the range of program content was positively 
associated with the total number of hours of youth participation after two and three years (r s =.16 
and r s =.32, respectively) and with the number of credits earned by high school participants after 
one and two years ( r s =.25 and r s =.24). The reasons for this relationship are unclear, although it 
may be an indication that providing too wide a range of activity offerings may appeal to youth 
but cause programs to spread their resources thin and thus be less likely to provide a focused and 
structured environment for participants. 



Relationships Involving Social Interactions 

In general, measures of a supportive OST environment, including the programs’ average 
youth reports of interactions with their peers and with staff members, were positively correlated 
with youth social and educational outcomes, as shown in Exhibit 3 1 . Most notably: 

■ Youth reports of their sense of belonging were positively correlated with the 
extent to which programs fostered positive interactions among youth after one, 
two, and three years of OST participation (r s =. 26, r v =.3 1 and r s =. 22, 
respectively). Similarly, sense of belonging was positively correlated with 
programs’ capacity to foster positive interactions between youth and staff, as 
measured by average youth reports of their interactions (r s =. 29 after one year, r s 
=.35 after two years, and r s =29 after three years). 

■ Youth also reported higher levels of academic benefits after one and two years of 
OST participation when they had attended a program that on average fostered 
positive interactions among youth (r v =. l 8 and r s = .24, respectively) and between 
youth and staff ( r s =.19 and r s =.26, respectively). 
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Exhibit 31 

Correlations between Features of Positive Relationships and Youth Outcomes 



Youth Outcomes 


Spearman Correlation Coefficient (rj 


Average Ratings of 
Youth Interactions 
with Peers 


Average Ratings 
of Youth 
Interactions with 
Staff 


Program Participation 


After 1 year 


-.29 


-.29 


After 2 years 


-.48 


-.43 


After 3 years 


-.45 


-.21 


Sense of Belonging 


After 1 year 


.26 


.29 


After 2 years 


.31 


.35 


After 3 years 


.22 


.29 


Prosocial Behavior 


After 1 year 






After 2 years 






After 3 years 


n/a 


n/a 


School Attendance 


After 1 year 




-.16 


After 2 years 




-.12 


After 3 years 






Academic Motivation 


After 1 year 






After 2 years 


.18 


.17 


After 3 years 






Academic Benefits 


After 1 year 


.18 


.19 


After 2 years 


.24 


.26 


After 3 years 






ELA Gains 


After 1 year 






After 2 years 


-.35 


-.22 


After 3 years 






Math Gains 


After 1 year 






After 2 years 






After 3 years 






Total Credits Earned 
(High School) 


After 1 year 


-.26 




After 2 years 






After 3 years 


n/a 


n/a 



Note: Only correlations that are statistically significant with effect sizes of at least .10 are included 
in this exhibit. Shaded cells indicate positive correlations. 



Exhibit reads: Total hours of participation after one year of OST enrollment and a program’s average youth 
reports of their interactions with peers were negatively correlated (r s =-. 29). 

The evaluation also found a few noteworthy patterns of negative associations between 
measures of positive relationships in the program and the level of youth attendance in the OST 
program. It is possible that this may be due in part to a lack of other alternatives available to 
youth, especially for younger participants, who do not have as much choice as their older peers 
to quit program participation if they are not happy with the program; however, there were too 
few high school and middle-grades programs in the evaluation’s in-depth sample to pennit this 
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analysis by program grade level. School attendance rates were also negatively correlated with 
average youth reports of their interactions with staff. One possibility for this pattern is that 
programs serving youth with the lowest school attendance rates are working in some of the most 
challenging environments, and these programs are attempting to build increased engagement in 
school by focusing on relationships in the program. 



Relationships Involving Effective Partnerships and Supports 

As illustrated in Exhibit 32, several measures of effective partnerships and supports were 
negatively associated with youth outcomes. For example, the number of professional 
development opportunities in which OST staff members participated was negatively correlated 
with youth program participation, reports of sense of belonging, and school attendance (r s 
ranging from -. 10 to -.29). Rather than being an indication that staff professional development is 
ineffective, more likely this finding is an indication that programs that know they are struggling 
to implement a high-quality program and contribute to positive youth outcomes are, in fact, 
taking greatest advantage of the various professional development opportunities available 
through DYCD, PASE, and other resources. 

Similarly, the extent of a program’s communication with schools and communication 
with families was negatively correlated with youth reports of their sense of belonging, school 
attendance, and (for parent communication only) perfonnance on the ELA state tests (r s ranging 
from -.10 to -.29 depending on the measure). In contrast, communication with schools and with 
parents was positively associated with the number of hours of youth participation in OST 
programming after one, two and three years (r s ranging from .26 to .37). The presence of a 
parent liaison was positively correlated with the level of youth participation after one, two, and 
three years (r pb =. 24, r pb =.25, and r pb =. 54, respectively). This suggests that OST programs work 
to fonn partnerships with schools and parents most when they are serving youth with the greatest 
academic needs. While the programs may not be contributing to immediate program gains, their 
outreach efforts are effective at enrolling and retaining participants. 

As described earlier, OST programs frequently employed a mix of staff members to 
provide services for youth, ranging from high school students to certified teachers or other 
professional specialists. Evaluators created an index variable that measured the number of 
different types of staff members employed by the program, and correlations revealed that having 
a variety of staff backgrounds was positively associated with participants’ sense of belonging in 
the program after one and two years (r s =. 1 1 and r s =. 14, respectively). However, school 
attendance rates were negatively correlated with programs’ patterns of hiring a diverse staff, 
perhaps because programs serving participants with low levels of school engagement are making 
efforts to reach out to them by hiring staff who differ from school-day staff 

Employing a master teacher was mixed in associations with youth outcomes. For 
example, master teachers were associated with more positive youth reports of their sense of 
belonging in the program after one, two and three years (r pb =. 22, r pb =. 22, and r pb =. 25, 
respectively) and with youth reports of academic benefits after one and two years (r s =. 1 9 and 
r s =. 17). However, there was a negative correlation between hiring a master teacher and the level 
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of youth participation in OST programming after one, two, and three years (r s =.-.\l, r s =.-. 29, and 
r s =.~. 14, respectively). Again, this suggests that OST programs may hire teachers more 
frequently when they are serving youth most in need of additional support and structure — and, in 
turn, the increased structure and outreach create a welcoming environment to improve youth 
belonging in the program. 



Exhibit 32 

Correlations between Features of Effective Partnerships and Supports 

and Youth Outcomes 



Youth Outcomes 


Spearman Correlation Coefficient (r s ) 


Mix of 
Staff 


Professional 

Development 


Communication 
with Schools 


Master 

Teacher* 


Communication 
with Parents 


Parent 

Liaison* 


Program 

Participation 


After 1 year 




-.26 


.26 


-.17 


.26 


.24 


After 2 years 


-.15 


-.18 


.33 


-.29 


.37 


.25 


After 3 years 


.16 




.32 


-.14 


.36 


.54 


Sense of 
Belonging 


After 1 year 


.11 


-.10 


-.10 


.22 


-.14 




After 2 years 


.14 


-.18 




.22 


-.21 


-.16 


After 3 years 




-.21 




.25 






Prosocial 

Behavior 


After 1 year 


.10 












After 2 years 














After 3 years 


n/a 


n/a 


n/a 


n/a 


n/a 


n/a 


School 

Attendance 


After 1 year 


-.13 






-.18 


-.11 


-.11 


After 2 years 


-.16 


-.12 


-.17 


-.11 


-.17 




After 3 years 


-.19 


-.29 


-.24 




-.27 




Academic 

Motivation 


After 1 year 




-.13 






-.10 




After 2 years 


.11 


.12 






-.16 


-.22 


After 3 years 














Academic 

Benefits 


After 1 year 


.19 






.19 




.19 


After 2 years 




-.22 




.17 


-.13 




After 3 years 














ELA Gains 


After 1 year 














After 2 years 


-.28 






-.25 


-.29 


-.31 


After 3 years 














Math Gains 


After 1 year 


.11 






.10 






After 2 years 














After 3 years 














Total 

Credits 

Earned 

(High 

School) 


After 1 year 


.27 


.27 


-.27 


.13 


.13 




After 2 years 














After 3 years 


n/a 


n/a 


n/a 


n/a 


n/a 


n/a 



* For dichotomous predictor variables, a point-biserial coefficient ( r pb ) is presented. 



Note: Only correlations that are statistically significant with effect sizes of at least .10 are included in this exhibit. Shaded 
cells indicate positive correlations. 

Exhibit reads: Total hours of participation after one year of OST enrollment and staff participation in professional 
development opportunities were negatively correlated (r s =-.26). 
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Program Quality Index 



Based on the findings from the correlation analyses described above, evaluators created a 
program quality index as a tool for assessing the overall quality of OST programs, using 
indicators available in this evaluation. The index is based on key structure, process, and content 
variables from the study’s participant survey and program-director survey, as displayed in 
Exhibit 33. Measures of program quality that consistently demonstrated a negative association 
with youth outcomes, such as the range of program content, communication with schools and 
families, and staff participation in professional development, were excluded from the index. 

Exhibit 33 

Components of the Program Quality Index (n=87) 



Component 


Source 


Mean 


Range 


Features of Rich Program Content 








Exposure to New Experiences 

(aggregated participant survey scale) 


Participant 

survey 


3.21 


1-4 


Features of Positive Relationships 








Youth Interactions with Peers 

(aggregated participant survey scale) 


Participant 

survey 


3.32 


1-4 


Youth Interactions with Staff 

(aggregated participant survey scale) 


Participant 

survey 


3.38 


1-4 


Features of Effective Partnerships and Supports 








Mix of Staff 

(count of the different types of staff employed) 


Director 

survey 


4.52 


1-7 


Master teacher 

(whether program has a master teacher) 


Director 

survey 


0.51 


0-1 


Parent liaison 

(whether program has a parent liaison) 


Director 

survey 


0.45 


0-1 



The program quality index was computed by summing a program’s standardized z scores 
on each of the components of the index. 9 The distribution of the program quality index, shown 
in Exhibit 34, demonstrates that OST programs varied in the extent to which they achieve a high 
level of overall program quality. 

As expected based on the correlations observed between individual measures of program 
quality and youth outcomes, analyses found moderate -to-strong positive correlation between 
overall program quality and aggregate reports of the following youth social and educational 
benefits: 



■ Youth reports of their sense of belonging in the program (r=.58) 

■ Youth reports of academic benefits (r=. 44) 



9 Because the variables within the program quality index are measured on different scales (for example, the exposure 
to new experiences scale ranges from 1 to 4, while the mix of staff scale ranges from 1 to 6), the evaluation 
computed z scores for each variable, and then added these z scores together to create a program’s quality index 
score. By converting the variables into z scores, which express values in terms of standard deviations, each element 
in the index is given the same weight. The evaluation calculated this program index for the 87 programs that had 
data for all of the variables in the index. 
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Youth reports of engagement in pro-social behaviors (r=. 37) 
Youth reports of academic motivation (r=. 32). 



Exhibit 34 

Distribution of OST Program Quality (n=87) 



20 



15 




Program Quality Index Score 




■ Program Quality Index (n=87) 





Exhibit reads: Approximately 3 percent of programs scored at the lowest end of the range of values for the program 
quality index. 



In addition, there was a strong and significant positive correlation between the program 
quality index and whether the program met its targeted enrollment (d=. 56). The strength of 
these relationships suggests that successfully implementing key features of program quality, 
including the provision of program content that exposes youth to a range of experiences and 
creation of an environment of positive relationships with diverse staff members, can contribute to 
an OST programs’ effects on the social and educational development of youth participants. 
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Multi-level Relationships 



To examine the relationships among program quality, youth participation, and youth 
outcomes, evaluators created a series of multi-level models. Multi-level modeling is a statistical 
method that accounts for the grouping of individuals within higher-level units. In this case, it 
allowed evaluators to examine OST participants within the context of their specific OST 
programs. 10 

Evaluators created a simple multi-level regression model for each youth engagement, 
social development, and educational development outcome, examining the impact of student- 
level participation (as measured by the total number of hours of exposure to OST programming) 
and of program-level quality (as measured by the program quality index) on the outcome. For 
educational performance outcomes measures gathered from DOE data, evaluators controlled for 
performance in the year prior to participation. In addition, the participants’ grade was controlled 
for in each model. 



Multilevel Model Equation: 

n=A,H (g,H «,)•* 

There is a total or composite residual that contains two parts for each participant: 

A participant-level residual, Ey 
A program-level residual, u ; . 

Predictors at each level are distinguished by subscripts: 

i = program 
j = participant 



Effects of Program Participation 

As shown in Exhibit 35, multi-level analyses revealed a similar pattern of positive 
relationships between OST participation and school attendance after one and two years of 
exposure to OST as was found in the correlational analyses. The low intra-class correlations 
(ICC) — which summarize the proportion of outcome variability due to differences across 
programs — for measures of school attendance after one and two years of OST participation 
(7CC=.l 1 and 7CC=.12, respectively) suggest that most of the variability in school attendance is 



10 By accounting for the shared program experience of youth participants within a program, multi-level modeling 
reduces error caused by violating the assumption of the independence of various youth outcomes, and allows 
evaluators to determine how much of the variability in outcomes can be attributed to program-level factors and how 
much can be attributed to participant-level factors (Miles & Shelvin, 2005). 



48 




due to differences within programs rather than across programs, and therefore participant-level 
differences (which were not measured by the evaluation) are likely the most important predictors 
of school attendance. 



Exhibit 35 

Directionality of Significant Relationships with Youth Outcomes 



Youth Outcome 11 


Youth 

Participation 


Program Quality 
Index 


Intra-Class 
Correlation (ICC) a 


Hours of Program Participation 








After 1 year of exposure 


N/A 


NS 15 


0.30 


After 2 years of exposure 


N/A 


C 


0.49 


Sense of Belonging 








After 1 year of exposure 


NS 


+ 


0.08 


After 2 years of exposure 


NS 


+ 


0.13 


School Attendance 








After 1 year of exposure 


+ 


NS 


0.11 


After 2 years of exposure 


+ 


NS 


0.12 


Academic Motivation 








After 1 year of exposure 


NS 


+ 


0.06 


After 2 years of exposure 


NS 


NS 


0.10 


Academic Benefits 








After 1 year of exposure 


NS 


+ 


0.12 


After 2 years of exposure 


NS 


+ 


0.11 I 


ELA Gain in Standardized Score 








After 1 year of exposure 


NS 


NS 


NS 


After 2 years of exposure 


NS 


NS 


0.13 


Math Gain in Standardized Score 








After 1 year of exposure 


NS 


+ 


0.03 


After 2 years of exposure 


NS 


NS 


NS 



a The intra-class correlation summarizes the proportion of the outcome variability that is due to differences across 

that the relationship was not statistically significant at the p<.05 level. 
c Because the program quality index was computed using standardized z-scored variables, the coefficients cannot 
be practically interpreted. Therefore, the exhibit emphasizes the direction of the significant relationships between 
the outcome and predictor variable, rather than the coefficient value. 



programs. 

“ NS indicates 



Effect of Program Quality 

As described above, many of the intra-class correlations for youth-reported outcomes 
were low, indicating that most of the variance in those outcomes was due to participant-level 
characteristics. Nonetheless, the program quality index was positively related to youth reports of 
their sense of belonging after one and two years of participation (7CC=.08 and ICC=. 13), 
suggesting that quality elements measured in the index made a small but significant contribution 
to participants’ social development. Similarly, in the multi-level models, program quality had a 
significant positive relationship with youth reports of academic benefits after one and two years 



1 1 Too few cases were available to permit the reporting of analyses of three years of exposure to OST. In addition, 
there were too few cases for analyses of the pro-social behaviors scale and of total credits earned in high school after 
two years of exposure. 
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of participation, although it explained only a small proportion of the differences between 
programs (7CC=.12 and ICC=. 11, respectively). While gains in math scores after one year of 
OST participation were positively related to the program quality index, the amount of variation 
in gains due to program-level characteristics was negligible (7CC=.03). This finding should be 
interpreted with caution given that the New York math test is administered in early March, after 
only a few months of OST programming. 

It is noteworthy that the intra-class correlations are relatively high for the measures of the 
total number of hours of program participation, meaning that differences across programs 
account for more of the variation. However, the multi-level analysis also revealed a significant 
negative relationship between the total number of hours of youth participation in OST after two 
years of program attendance and the program quality index (7CC=.49). One possibility for this 
relationship is that, while the program quality index measures features of programs that research 
has shown may contribute to better youth educational and social outcomes, youth themselves 
may be attracted to programs for a variety of other reasons not measured by the index. As noted 
earlier in this report, youth are overwhelmingly positive in their reactions to the OST initiative 
and report high levels of satisfaction with the programs and the safe spaces they provide. 
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6. The OST Initiative’s Role and Impact on 
Creation of a City-wide OST System 



This chapter reviews the contribution of the OST initiative to three key components of 
system development, including improvement of the capacity of provider organizations, 
professionalization of the out-of-school time youth-service field, and satisfaction of the needs of 
working families. 



Capacity of Provider Organizations 

Through the OST initiative, provider organizations in New York City have increased 
their capacity to serve large numbers of youth during the out-of-school time hours. As described 
earlier, the number of youth served under the initiative increased from about 5 1,000 in the first 
year to more than 81,000 in the third year. In addition, in the third year of the initiative, 111 new 
elementary-grades programs received funding to offer school-year and summer programming. 

Executive directors of provider organizations reported that the OST initiative increased 
their organizational capacity in several ways, as shown in Exhibit 36. In Year 3, more than half 
of executive directors reported that the initiative increased the organization’s capacity to serve 
more youth and families to a great extent or somewhat (83 percent), to provide staff training and 
technical assistance (73 percent), partner with a public school (71 percent), partner with cultural 
organizations (65 percent), partner with city agencies (63 percent), offer programming on 
weekends and holidays (59 percent), and provide a career ladder for OST staff (57 percent). 
These patterns of responses were similar to those from the first two years of the evaluation. 
However, in Year 3, reflecting the OST initiative’s maturity in the schools and community, 
executive directors reported significant increases in their ability to partner with public schools 
and to partner with cultural organizations, as compared to Year 1. The percent of executive 
directors who reported that the OST initiative increased their organizational capacity to partner 
with a public school increased from 59 percent in Year 1 of the OST initiative to 71 percent in 
Year 3 (V=.\ 3). Similarly, the percent of directors who reported increased capacity to partner 
with cultural organizations rose from 55 percent in Year 1 to 65 percent in Year 3 (V=. 1 0). 
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Exhibit 36 

Executive Director Reports of Increased Capacity, in Percents (n=169) 



Serve mo re yo uth an d fami lies 
Partner with a public school 



Offer p ro grammi ng on weekends an d 
holidays 



Provide staff training and technical 
assistance 



Partnerwith cultural organizations 
Partner with city agencies 
ProvideacareerladderforOST staff 



Access city funding other than DYCD 



Access state fun ding | 6 | 33 



Access corporate/private funding N 30 



Access federal fun ding 2 19 




20 



40 



60 



80 



100 



ITo a great extent 



iSomewhat 



Exhibit reads: Fifty-three percent of executive directors agreed to a great extent and 30 percent somewhat agreed 
that participating in the OST initiative had increased their organization’s capacity to serve more youth and families. 

Executive directors were also asked to rate the extent to which their OST programs 
implemented certain practices in the third year of the OST initiative, compared to previous years, 
as shown in Exhibit 37. Fifty-nine percent of executive directors reported that their programs 
provided much more or somewhat more training and technical assistance for staff in the third 
year of the initiative than they did in the first or second year. In addition, nearly half reported 
that programs were better able to establish linkages with schools surrounding their programs (48 
percent), enforce minimum attendance policies for participants (47 percent), and track student 
program attendance (46 percent). 



52 



Exhibit 37 

Reports of OST Implementation Compared to Previous Years, in Percents (n=160) 



Provide trai ning and technical ass istance for staff 
Establishlinkageswithsu rr oundi ng s chools 
Enforce minimum attendance policies for participants 

Track student program attendance 

Adhere to strict standards about hiring and screening 
qualified staff 

Comply with city an d state chi Id-ca re reg ul ations 
Offer p ro gramm i ng on weekends an d h oli days 
Offersummerprograms 
Offer regular, extensive programming hours 
Have a lowstaff-to-youthratio 
Provide access for children withspecial needs 

0 20 40 60 80 100 

□ Responded "Much mo re" or 'Somewhat more" 

Exhibit reads: Fifty-nine percent of executive directors surveyed reported that their programs provided much more or 
somewhat more training and technical assistance for staff this year, compared to previous years. 




Overall, executive directors reported high levels of satisfaction with DYCD’s 
management of the OST initiative and support of OST programs, as shown in Exhibit 38. More 
than half of executive directors reported that they were very satisfied with opportunities offered 
for staff professional development (55 percent). In addition, about two-thirds (67 percent) of 
executive directors were very satisfied with the support provided by the DYCD program 
manager, although site-level program directors are more likely to interact directly with DYCD 
managers than are executive directors. Executive directors were least satisfied with DYCD’s 
assistance in negotiating partnerships with schools (29 percent reported that they were very 
satisfied). 
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Exhibit 38 

Executive Director Reports of Satisfaction, in Percents (n=168) 



Support provided by 
DYCD program manager 



Opportunities for staff 
professional 
development 



Mo n itoring of en rollment 
and participation 



Implementation of DYCD 
Online management 
information system 



Tec h n ical as si stance i n 
program design and 
implementation 



Assistance negotiating 
partnerships with schools 



0 


20 


40 


60 




■ Very satisfied 




DSomewhat satisfied 



Exhibit reads: Sixty-seven percent of executive directors surveyed were very satisfied and 27 percent were 
somewhat satisfied with the support provided by DYCD program managers. 




In general, executive directors reported few significant challenges to implementing out- 
of-school time programming through the OST initiative. The most frequent challenges reported 
included administrative burden (48 percent reported that this was a challenge to a great extent or 
somewhat), meeting enrollment and attendance requirements (44 percent), and using the DYCD 
Online tracking system (39 percent). These challenges were similar to those reported by 
executive directors during the first two years of the OST initiative. However, there were some 
notable changes in challenges reported. The percent of executive directors reporting 
administrative burden as a challenge decreased from 61 percent in Year 1 to 48 percent in Year 3 
(V=.13), and the percent reporting that using DYCD Online was a challenge decreased from 54 
percent to 39 percent (V=.15), reflecting the system’s learning curve. In contrast, the percent of 
executive directors reporting that hiring qualified staff was a challenge increased from 23 percent 
in Year 1 to 33 percent in Year 3. (V=.10). 
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Professionalizing the Field of Out-of-School Time 



On-site OST program directors received relatively high levels of professional benefits, as 
reported by executive directors of provider organizations. More than three-quarters of executive 
directors reported that they offered program directors paid training or professional development 
(85 percent), paid attendance at staff meetings and conferences (85 percent), paid time off for 
vacation and sick leave (83 percent), and health insurance (78 percent). This reflected a stable 
level of benefits in comparison to the first two years of the initiative. 

In contrast to program directors, OST program staff received few benefits outside of 
professional development. Nearly three-quarters of provider organizations offered OST staff 
paid attendance at staff meetings and conferences (72 percent), and 69 percent offered paid 
training or professional development. The percent of organizations offering paid attendance at 
staff meetings represented a notable increase from Y ear 2 of the initiative, when only 62 percent 
of executive directors reported this benefit for staff (V=. 1 1). 



Meeting the Needs of Working Families 

A goal of the OST initiative was to provide support to working families in New York 
City, particularly in the target zip codes identified as priorities for out-of-school time services. 

In Year 3, as in previous years, survey responses from parents of OST participants in the 
elementary and middle grades indicated that overall the initiative succeeded in reaching this goal 
and meeting the needs of families. As shown in Exhibit 39, about three-quarters of parents rated 
the OST program that their child attended as either excellent (43 percent) or very good (33 
percent). 

As in previous years of the evaluation, parents especially valued the academic support 
features of OST programs in survey responses. Forty-seven percent of parents cited homework 
help as the most important activity in the after-school program, and an additional 26 percent 
cited academic enrichment as the most important activity. In addition, as illustrated in Exhibit 
40, parents’ reports of their reasons for enrolling their child in the OST program reflected an 
emphasis on seeking academic support: 76 percent believed the program would help their child 
do better in school, and 72 percent wanted their child to get help with homework. Seventy-four 
percent of parents also said that they enrolled their child in an OST program to provide them 
with the opportunity to participate in new activities. 
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Exhibit 39 

Parent Ratings of the OST Program, Year 3 (n=413) 




Exhibit reads: Forty-three percent of parents rated the OST program that their child attended as excellent. 



Exhibit 40 

Parent Reports of Reasons for Enrolling Their Child, Year 3 (n=407) 



I believed program w ould help my child do better in school 
I wanted my child to participate in new types of activities 
I w anted my child to get help w ith homew ork 
I w anted my child to be safe after school 
My child w anted to enroll 
I wanted my child to develop better personal/social skills 
I wanted my child to gain more confidence in himself /herself 

The program is free 
I needed childcare 
It w as a w ay for my child to be w ith friends 

0 20 40 60 80 10n 

Percent who cited "a big reason" 

Exhibit reads: Seventy-six percent of parents surveyed enrolled their child because they believed the program 
would help their child do better in school. 
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Overall, parents reported satisfaction with the ways in which the OST program supported 
their children academically. As illustrated in Exhibit 41, 91 percent of parents agreed that their 
child is doing better in school as a result of participation in OST. Similarly, 90 percent agreed 
that their child was getting the academic help he or she needed. 



Exhibit 41 

Parent Reports of Youth Benefits of Participation, Year 3 (n=406) 



My child is able to join activities that he/she would not be able to 
otherwise attend 

Mychildissaferin the out-of-school hours 
Mychildis making new friends 
My child is doing better in school overall 
My child talksto me mo re about what's going on in schools 
My child is mo reconfident 
My child is getting needed academichelp 
My child is behaving better at home 




■ Strongly Agreed 



D Agreed 



Exhibit reads: Forty-seven percent of parents surveyed strongly agreed that because of their OST program, their child 
is able to join activities that he or she would not be able otherwise to attend. 



[The program has] helped my son with his homework, with his ability to learn how to 
socialize and be more outspoken with people and strangers. It has also helped him learn 
more about art, chess, and other board games, for example, chess, checkers, Scrabble, 
etc. 

This program helped my son and my family in many ways. My son gets help with his 
daily homework. He gets to be with his friends and participate in other after-school 
activities. It also allows my wife and me to work a full day without worrying about after- 
school hours care for my son. It is a great program for my son and my family. 

The availability of this program has helped us so much. It has helped my son to integrate 
with more friends, participate in various activities offered by the program, and also it 
allows him to go over his school matters and do his homework with the help of the great 
staff. They are all wonderful and superb, and most of all this program allows me and my 
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wife to be able to attend our jobs without worrying about how we’re gonna handle [our 
son ] after school. 

My child is very active and very curious intellectually. This program gives him an outlet 
and a source of knowledge and keeps him busy. 

Well, it’s helped [my son] with his studies, because sometimes I don ’t know enough to 
help him with his homework. [Translated from Spanish.] 

As shown in the exhibits above, parents also reported that they chose to enroll their 
children in an OST program to support their social development. Sixty-eight percent of parents 
said that they wanted their child to develop better personal and social skills, 64 percent wanted 
their child to gain more confidence, and 48 percent felt that the OST program was a way for their 
child to be with friends after school. Parents agreed that the OST program fulfilled these goals: 
95 percent agreed that their child was making new friends, and 92 percent agreed that their child 
was more confident as a result of participating. 

My child is sociable, she’s learning more, and she’s confident about herself. School 
work is getting done and her attitude about going to school changed — she likes school 
now. 

[My child’s] work is school has improved greatly as well as his self-esteem. He is more 
outgoing and is very sure of himself in the things that he feels he can achieve. 

My son ’s teachers suggested that I enroll him in the program. Now he expresses himself 
better because program activities have endowed him with more self-confidence, and he ’s 
become more sociable and better-behaved in group activities. [Translated from 
Spanish.] 

I think highly of the program, because my children are developing higher physical and 
mental skills. In addition, they ’ve learned how better to relate with other children and 
have self-confidence. [Translated from Spanish.] 

Finally, parents reported enrolling their children in the OST program because it offered 
them a safe place to be after school (7 1 percent), was free of charge (63 percent), and offered 
needed child care (54 percent). Indeed, parents overwhelmingly reported that they worked or 
attended school. The majority of parents responding to the survey reported that they worked 
outside the home (94 percent), including 2 1 percent who worked more than 20 hours per week. 

In addition, 84 percent of parents reported that they were pursuing their own education (86 
percent of this group attended school less than 10 hours per week). Parent comments reflected 
the value they placed on the OST program for allowing them opportunities to work or pursue 
their education more than they would have otherwise. 

In interviews and on surveys, parents commented: 
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The program is helping my family a lot because 1 am able to attend college full time... I 
am very thankful to the program, and it’s very essential to have those kinds of programs 
in public school. 

Thanks to the program, my wife has the chance to have a job. Besides, my daughter feels 
very comfortable during the program because she is around children her age. 

I am a single parent and having three of my children in this program made it possible for 
me to be able to go back to work full-time. 

I was able to go back to school and my husband was able to work more. My child is 
safer in the after-school program than staying with a babysitter. 

This free after-school program provided a safe and affordable place for my son. The 
program enabled me to work a full day. 

I think that this after-school program has literally saved my life. If it wasn ’tfor [ the 
program], I don ’t know where I would’ve been. So thanks to this program I’m able to 
attend college and make myself a better future for my life. 

[The program] cdlows me to work longer hours so I can help economically at home and 
much more. [Translated from Spanish.] 

In genercd, I think the program is a great help for parents because while they’re working, 
they know that their children are being looked after and not alone at home and they can 
participate in many fulfilling and worthwhile activities. [Translated from Spanish.] 

Parent responses about the benefits of the OST program also confirmed the sense of 
security provided: 85 percent agreed that their child was safer in the out-of-school hours as a 
result of the program. 

Parents also responded positively to questions about the ways in which the OST initiative 
had enabled them to work more or pursue more education. Across all responding parents, as 
shown in Exhibit 42, 74 percent agreed that the program made it easier for them to keep their 
job, and 73 percent agreed that they miss less work than they had previously because their 
children attended the OST program. In addition, 7 1 percent of parents reported that they were 
able to work more hours because their children were in the program. 
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Exhibit 42 

Parent Reports of Family Benefits of Participation, Year 3 (n=408) 



The program hours fit my needs 



The program has made it easierto 
keep my job 



I miss less workthan I used to 



I am able to work more hours 



I am able to go back to work 



The program has made it easier 
for me to go to school 



0 20 40 60 80 100 



■ Strongly Agreed DAgreed 



Exhibit reads: Fifty-seven percent of parents surveyed strongly agreed that their program’s hours fit their needs. 
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7. Looking Ahead to Long-Term Sustainability 



Through the first three years of the OST initiative, DYCD has established the policy and 
practical foundations for the long-tenn sustainability of high-quality, publicly funded, out-of- 
school time programming for the youth of New York City. With the launch of the OST initiative 
in 2005, the collaborative relationship between DYCD and the New York City Department of 
Education has grown, particularly through adoption of a Memorandum of Understanding 
committing hundreds of public schools as sites for OST programs and the provision of in-kind 
support by DOE for OST programs through facilities, security, and snacks and meals for OST 
participants. At the same time, DYCD strengthened its network of community-based 
organizations, foundations, and providers of technical assistance to support the initiative through 
partnerships with the Wallace Foundation and the Partnership for After-School Education, 
among others. Importantly, the City has included funding for OST programming in its four-year 
financial plan, and the budget for OST programming has steadily increased from $46.4 million in 
FY 2006 to $117.1 million in FY 2009. 

As the OST initiative enters its next phase, evaluation findings from the first three years 
point to elements of program quality that should be maintained as well as areas in which focused 
resources and technical assistance can improve quality and both educational and social outcomes 
for youth. 

Engaging youth. Over the first three years of the OST initiative, out-of-school time 
programs in New York City succeeded in reaching large numbers of youth, serving more than 
181,000 participants, including 7,589 participants who remained enrolled for three years. The 
initiative scaled up rapidly from 2005-06 to 2007-08, from 50,618 participants in 528 programs 
in the first year of programming to 81,213 participants in 622 programs in the third year. 
However, some programs continued to struggle to enroll high numbers of participants 
(approximately one -third of programs did not meet their enrollment target) and to engage all 
participants at high levels of attendance (fewer than half of program participants achieved the 
targeted number of hours of participation). 

Evaluation findings point to positive associations between high levels of participation and 
youth’s sense of belonging in the program and academic benefits, suggesting that directing 
efforts to help programs increase their appeal to youth and better engage participants can lead to 
long-term benefits. The evaluation also found that effective strategies for achieving high levels 
of participation include offering a range of activities for youth and communicating regularly with 
schools and with families. 

Creating positive environments. Overall, OST programs succeeded in their objectives to 
create positive environments for youth by fostering healthy relationships among youth 
participants and between youth and staff members. Programs achieved this goal in varied ways, 
including the implementation of activity structures that focused on relationship-building and 
development of social skills and by hiring a diverse mix of staff members, including younger 
staff who could relate to and serve as mentors and role models for youth. As a result, youth 
reports of their interactions were overwhelmingly positive. OST programs faced more 
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challenges in implementing active learning experiences for youth, as evidenced through survey 
data as well as program observations. 

Evaluation data also suggest that, as they scaled up programming and become 
increasingly established within their schools and communities, OST program directors took 
advantage of professional development opportunities — both internal and external — and built 
strong partnerships with their schools and communities, in order to improve OST program 
quality and better meet the needs of the youth served. 

DYCD has committed resources to improving the quality of its monitoring and support of 
OST programs through ongoing technical assistance opportunities, particularly those focused on 
data management, behavior management techniques, and program content development. The 
DYCD Online data management system offers opportunities to continue to track patterns of 
program and activity engagement across the initiative and within specific programs. In addition, 
the development of new rubrics and program management tools, including those that track the 
implementation of features in the evaluation’s program quality index, will provide opportunities 
to continue to strengthen the capacity of OST programs to provide high-quality services to youth. 

Shaping the city’s OST system of the future. Based on the evaluation findings from the 
first three years of the OST initiative, we offer the following additional recommendations for 
DYCD in improving the capacity for OST programs to deliver high-quality services to improve 
outcomes for New York City’s youth: 

■ Assist programs in identifying resources — or in learning to better plan and budget 
existing resources — directed specifically to hiring specialized staff members to 
maximize youth recruitment and engagement (e.g., parent liaisons) and to help 
plan and oversee high-quality, structured program content (e.g., certified teachers 
or professional specialists). 

■ Focus technical assistance related to activity planning on teaching staff strategies 
to engage youth in dynamic, active learning opportunities in which they discuss, 
collaborate, plan, and take on leadership roles, regardless of the content area. 

■ Through technical assistance, encourage OST programs and provider 
organizations to utilize the capacity of the DYCD Online system to generate data 
that can support program management and improvement efforts, including, for 
example, the monitoring of program participation patterns to determine whether 
certain types of activities appeal more or less to particular groups of students. 
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Appendix A 



Survey and Observational Data Collected 
in Years 1-3 of the OST Initiative 



Exhibit A-1 

Executive Director Survey Data 





Executive Director Surveys 




Number of 
Executive 
Directors 


Number of 
Surveys 
Received 


Response Rate 
(in percents) 


Year 1 


190 


161 


85 


Year 2 


191 


148 


77 


Year 3 


203 


169 


83 



Exhibit A-2 

Program Director Survey Data 





Programs Director Surveys 


Number of 
Program 
Directors 


Number of 
Surveys 
Received 


Response 

Rate 

(in percents) 


Option 1 
Surveys 
Received 


Option II 
Surveys 
Received 


Option III 
Surveys 
Received 


Year 1 


543 


483 


89 


393 


80 


10 


Year 2 


547 


470 


86 


385 


77 


8 


Year 3 


630 


555 


88 


460 


87 


8 



Exhibit A-3 

Participant Survey Data 



Year 


Participant Survey Sample (n=118 programs) 


In-depth Sample (n=15 programs) 


Surveys 

Received 


Programs Returning 
Surveys 


Surveys 

Received 


Programs Returning 
Surveys 


Year 1 


3,088 


82 


524 


13 


Year 2 


4,499 


86 


837 


14 


Year 3 


5,490 


93 


811 


15 
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Exhibit A-4 

Program Staff and Parent Survey Data from 15 In-Depth Sites 





Program Staff Surveys 


Parent Surveys 1 




Surveys 

Received 


Programs 

Returning 

Surveys 


Surveys 

Received 


Programs 

Returning 

Surveys 


Year 1 


114 


12 


283 


12 


Year 2 


191 


13 


500 


12 


Year 3 


193 


14 


450 


13 



Exhibit A-5 

Site Visit Observation Data 





Independent 

Observations 


Co-observations 


Inter-rater 

Reliability 


Year 1 


238 (in 15 programs) 


40 


0.83 


Year 2 


199 (in 15 programs) 


33 


0.72 


Year 3* 


141 (in 12 programs) 


10 


0.76 



*ln Year 3, the evaluation emphasized qualitative data collection and observations in OST 
high school programs, which reduced the number of programs included in observation data 
analyses. 
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Appendix B 

Program Implementation in Options II and III 



The DYCD OST initiative supports programming under three service options. The focus 
of the evaluation is Option I programs, which serve youth in elementary, middle, and high 
schools throughout New York City. In addition, Option II programs were designed to build on 
public-private partnerships and were required to receive at least 30 percent of their funding from 
private sources such as corporations, foundations, and individuals. Option III programs operate 
through the Department of Parks and Recreation and are offered at Parks sites. 

In Year 3, the evaluation collected program director survey data for 86 Option II 
programs and eight Option III programs. DYCD Online data were available for 12,340 
participants in 96 Option II programs, and 1,349 participants in 12 Option III programs for Fiscal 
Year 2008 (July 2007 through June 2008). Data were also available for 5 1 Option II programs 
and one Option III program for July of 2008. Because of the different structures and 
expectations of Option II programs and Option III programs, evaluators analyzed their data 
separately. This appendix presents a summary of program implementation under these service 
options in Year 3 of the initiative. 



Scope of OST Services 

In Fiscal Year 2008, Options II and III served almost 14,000 youth in 108 programs 
across New York City. " As shown in Exhibit B-l, programs operated in all boroughs and in all 
grade levels, with the most Option II participants in Manhattan and the most Option III 
participants in Brooklyn. 

Eighty percent of Option II program directors reported that they have open enrollment for 
all interested youth. Fifty-nine percent of Option II programs also reported that they seek to 
serve youth who were recommended by school-day teachers or counselors. 

All eight responding Option III program directors reported that they have open 
enrollment for all interested youth. More than half of Option III programs also reported seeking 
to serve youth who scored below proficient on city or state exams (6 programs), youth with 
siblings already attending the program (7 programs), and youth who participate in other 
programs sponsored by the organization (7 programs). 



12 Because DYCD tracks their Option II and III programs according to fiscal years, the numbers for Option II and III 
programs represent the time period July 2007 through June 2008. 
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Exhibit B-1 

FY 2008 Number of OST Programs and Participants, by Option 



Program 

Characteristics 


Option II 


Option III ] 


Programs 

(n=96) 


Participants 
(n=1 2,340) 


Programs 

(n=12) 


Participants 

{n=1,349) 


Borough 










Brooklyn 


24 


3,313 

(27) 


3 


578 

(43) 


Bronx 


23 


2,352 

(19) 


2 


68 

(5) 


Manhattan 


31 


3,564 

(29) 


4 


221 

(16) ! 


Queens 


16 


3,016 

(24) 


2 


257 

(19) 


Staten Island 


2 


95 

(D 


1 


225 

(17) 


Program Location 










School 


40 




N/A 


N/A 


Center 


56 




12 


1,349 


School Level 










Elementary 


42 


6,695 

(54) 


8 


866 

(64) 


Middle 


19 


2,410 

(20) 


2 


428 

(32) 


High 


35 


3,235 

(26) 


2 


55 

(4) 



Figures in parentheses indicate the percent of participants with each characteristic within each category. 
Percents do not necessarily add to 100 due to rounding. 

Exhibit reads: Twenty-four out of the 96 Option II programs in FY 2008 were located in Brooklyn, as were 
3,313 of participants, who made up 27 percent of all Option II participants. 



As shown in Exhibit B-2, Options II and III programs served youth in a range of grade 
levels, with the highest concentration of youth in elementary grades. The gender ratio of youth 
served in Option II was balanced, but Option III programs served more males than females (63 
percent, compared to 38 percent.) Youth in Options II and III programs came from diverse 
backgrounds: 42 percent of youth in Option II and 36 percent of youth in Option III were 
Hispanic/Latino, and 30 percent of Option II youth and 26 percent of Option III youth were 
African American. 
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Exhibit B-2 

Demographic Characteristics of Participants, by Option, in Percents 



Total Number of Enrolled 
Participants 


Option II 


Option III 


n=1 2,340 


n= 1,349 


Grade Span 


n=1 2,340 


n= 1,349 


K-5 


54 


64 


6-8 


20 


32 


9-12 


26 


4 


Gender 


n= 12,340 


n= 1,349 


Male 


49 


63 


Female 


51 


38 


Race/ethnicity 


n=1 2,340 


n= 1,349 


American Indian 


0 


0 


Asian 


8 


4 


African American 


30 


26 


Flispanic/Latino 


42 


36 


Pacific Islander 


0 


0 


White (non-Hispanic) 


10 


8 


Other 


10 


26 



Exhibit reads: Fifty-four percent of participants in Option II were in grades K through 5. 



Evidence of Program Quality 

In the Year 2 report, evaluators identified certain elements of OST programs that may be 
conducive to producing a high quality program. Program director surveys from Option II and III 
programs addressed some of these elements: whether programs presented rich, varied content; 
whether staff were deployed effectively and well supported; and whether effective program 
partnerships and supports were in place. 



Rich Program Content 

Option II program directors most frequently reported social development goals for their 
programs. In particular, they reported that a major objective of their program was to provide a 
safe environment for youth (94 percent). Other major objectives included: 

■ Help youth develop socially (89 percent) 

■ Promote respect for diversity (8 1 percent) 

■ Provide health/well-being/life skills development (81 percent) 

■ Provide leadership opportunities (77 percent) 

Option III program directors reported a mix of academics and social development goals. 
All eight responding Option III program directors reported that major objectives of their program 
were to provide a safe environment for youth, help youth improve their academic perfonnance, 
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help youth develop socially, promote respect for diversity, and provide hands-on enrichment 
activities. 



Evaluators analyzed the activities that program directors reported offering to all or most 
youth. More than three-quarters of Option II program directors reported offering the following 
activities to the majority of participants on an ongoing basis: unstructured time for socializing 
(98 percent), peer discussion of topics that are important to youth (90 percent), and discussion 
about diversity issues (85 percent). 

Option III program directors most frequently reported offering the following activities to 
most participants on an ongoing basis: homework help (seven programs), discussion of issues, 
events, or problems in their community (five programs), group instruction in specific academic 
subjects (four programs), recreational reading (five programs), organized writing activities (three 
programs), and unstructured time for socializing (five programs). 



Staffing Strategies 

Option II programs had a median youth- to-staff ratio of 8: 1, including both paid and 
volunteer staff members. Sixty-seven percent of Option II program directors reported between 1 
and 10 paid staff members, and 43 percent reported between 1 and 10 volunteer staff. Staff came 
from many backgrounds: 55 percent of program directors reported employing specialists, 65 
percent employed college students, 33 percent employed high school staff, and 29 percent 
employed certified teachers. Seventy-eight percent of Option II programs reported hiring a staff 
member either part- or full-time to provide administrative support, and approximately half of 
programs employed a master teacher (48 percent). 

Seven of eight responding Option III program directors reported between one and 10 paid 
staff members, and only one program had volunteer staff. The median youth-to-staff ratio in the 
eight Option III programs was 19:1. Six programs employed college students, and six programs 
employed high school staff. Three programs had a part-time paid administrative support 
position, and three Option III programs employed a master teacher on either a part-time or full- 
time basis. 

Building staff capacity. Option II program directors reported frequent internal 
supervision and training opportunities. Nearly all Option II program directors (98 percent) 
reported holding staff meetings at least monthly, and 52 percent of program directors held staff 
meetings at least once a week. Seventy-two percent of program directors required most or all 
staff to submit activity plans on a regular basis, and 26 percent of programs used a published or 
externally developed curriculum to guide at least some of their activities. 

All eight Option III program directors reported holding staff meetings at least monthly, 
while three program directors held staff meetings at least once a week. Two program directors 
required most or all staff to submit activity plans on a regular basis, and another two program 
directors occasionally asked staff to submit activity plans. None of the program directors used a 
published or externally developed curriculum to guide any activities. 
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Seventy-seven percent of Option II program directors reported participating in workshops 
offered through the OST initiative, 47 percent participated in institutes or conferences, and 3 1 
percent participated in on-site consultations. The topics on which nearly half of program 
directors received training or professional development were program development and 
management (48 percent), maintaining healthy and safe environments (47 percent), and using 
developmentally appropriate practices (53 percent). Program directors reported that their staff 
received similar types of training in similar topics. 

Among the training and professional development activities offered to Option III 
program directors through the OST initiative, seven reported participating in workshops, three 
participated in institutes or conferences, and one participated in on-site consultations. The topics 
on which half or more of program directors received training or professional development were 
academics, enrichment, and learning (four programs), maintaining healthy and safe environments 
(five programs), and program development and management (five programs). 

Staff challenges. Forty-eight percent of Option II program directors reported that over 
half of their program staff worked in the same OST program during the previous year. However, 
hiring and adequately compensating qualified staff remained a challenge for most Option II 
program directors. More than half of Option II program directors reported the following 
staffing-related obstacles to implementing high-quality programming: 

■ Capacity to offer the competitive salaries necessary to hire qualified staff (72 
percent) 

■ Finding volunteers with the time and expertise needed (59 percent) 

■ Not being able to afford to offer potential staff enough hours of paid employment 
(55 percent) 

Two of eight Option III program directors reported that more than half of their program 
staff remained in their program in Year 3. Among Option III program directors, more than half 
reported that not being able to afford to offer potential staff enough hours of paid employment 
and offering the competitive salaries necessary to hire qualified staff were challenges to 
implementing high quality programming in Year 3. 



Effective Program Partnerships and Supports 

Establishing policies and structures. Option II program directors reported establishing 
effective program policies and structures in the third year of the initiative. Ninety-seven percent 
of program directors strongly agreed or agreed that the time allowed for activities in their 
programs was generally appropriate. Three-quarters or more of Option II program directors 
strongly agreed or agreed that groups were small enough for staff to meet participant/individual 
needs (97 percent), procedures were in place to report suspicions of child abuse and neglect (93 
percent), the program had a process in place for obtaining participant input and suggestions (93 
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percent), procedures for dealing with participant behavior were in place and effective (92 
percent), the program had links to organizations to which they could refer participants in need of 
additional services (89 percent), participants had regular opportunities to lead activities (8 1 
percent), and participants with special needs were successfully integrated (80 percent). 

Similarly, directors of Option III programs reported effective program structures. Four of 
the seven responding Option III program directors reported discussing the needs or progress of 
individual students with school principals. Two program directors reported talking with teachers 
or other key staff at least monthly, and two regularly discussed homework assignments. Five 
Option III program directors reported that at least one other outside organization provided 
activities or services for participants, and three reported that other organizations provided 
additional funding through grants or contracts. 

Option III program directors most commonly reported challenges related to youth 
recruitment or participation. Six of eight program directors reported that a major or minor 
challenge to implementing high-quality programming was that youth do not attend the OST 
program regularly enough to have enriching experiences, and six reported that they cannot 
recruit enough youth to participate. Four of six responding Option III program directors strongly 
agreed or agreed that, compared to Year 2, the program did a better job of fostering positive 
relationships between youth and staff in the third year. 

Partnerships with parents. Seventy-nine percent of Option II program directors reported 
that supporting working families was a major or minor objective, as did seven of eight 
responding Option III directors. 

Eighty-seven percent of Option II program directors reported having conversations with 
parents over the phone a few times a month or more. Seventy-eight percent of program directors 
reported meeting with one or more parents a few times a month or more. Program directors also 
reported doing the following a few times a month: sending material about the program home to 
parents (42 percent), holding events or meetings to which parents were invited (41 percent), and 
holding events or meetings to which community members were invited (24 percent). At least 
half of Option II program directors reported that their program sponsored the following events or 
activities for parents/families at least monthly: events at the program (84 percent) and 
opportunities to attend cultural or recreational events in the community (67 percent). 

Seven of eight Option III program directors reported having conversations with parents 
over the phone at least a few times a month. Six of seven program directors reported meeting 
with one or more parents at least a few times a month. Five of seven responding Option III 
program directors reported offering families opportunities to attend cultural or recreational 
events in the community at least monthly, and six regularly invited parents to events at the 
program. 
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Evidence of Youth Outcomes 



As shown in Exhibit B-3, Option II programs at all grade levels exceeded their 
enrollment targets, as did Option III middle-grades programs. Analyzed by program, 7 1 percent 
of Option II programs and 46 percent of Option III programs met or exceeded their enrollment 
targets in the third year of the OST initiative. 

Exhibit B-3 

Targeted Enrollment and Actual Number of Students Served, 
by Option and Grade Level 



Grade Level 


Option II 


Option III | 


Targeted 

Enrollment 


Students 

Served 


Percent 
of Target 
Achieved 


Targeted 

Enrollment 


Students 

Served 


Percent of 
Target 
Achieved 


Elementary 


5,916 


6,695 


113 


1,025 


866 


84 


Middle 


1,995 


2,410 


121 


275 


428 


156 


High 


3,118 


3,235 


104 


225 


55 


24 


Total 


11,029 


12,340 


112 


1,525 


1,349 


88 



Exhibit reads: In Option II, the total targeted enrollment for elementary programs was 5,916, but elementary 



programs actually served a total of 6,695 youth (or 1 13 percent of the target.) 



Option II programs were expected to offer at least 160 hours of programming, with a 70 
percent rate of participation. On average, Option II participants attended 184 hours of 
programming in Year 3, exceeding this goal. The median number of hours participants attended 
was 150. Option III programs were expected to offer 144 to 360 hours of service, depending on 
the youth served. However, on average Option III participants attended only 76 hours of 
programming during FY 2008; the median number of hours attended in Option III was 53. 
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Appendix C 

Technical Properties of Participant Survey Scales 



For each series of survey items addressing a common theme, evaluators created a survey scale to 
measure participants’ overall response to that theme. For each of the individual survey items, 
participants were asked to respond whether they agreed a lot, agreed a little, disagreed a little, or 
disagreed a lot. These survey items were combined into a scale calculated to range from 1 to 4, 
with 4 indicating that on average participants agreed a lot with each of these statements, and 1 
indicating that on average participants disagreed a lot with each of these statements. 

This appendix describes the individual items that are included in each scale, and presents the 
following statistical properties of each scale: 

■ Cronbach’s Alpha — a measure of the internal consistency of the survey scale 
ranging from 0-1, with higher numbers indicating a greater cohesiveness of items 

■ Mean — the average score on the scale across all participants, ranging from 1 to 4 

■ Standard deviation — an estimate of the average variability of the scale data 

■ Minimum/maximum — the minimum and maximum scores possible on the scale 

■ 25 th percentile/75 th percentile — respectively, the scale scores below which 25 
percent of participants and 75 percent of participants scored 



Academic Benefits of the Program 

The Academic Benefits of the Program scale was computed to range from one to four, with four 
indicating that on average participants strongly agreed with the following statements: 

This program has helped me. . . 

■ Get better grades in school 

■ Feel better about my schoolwork 

■ Read and understand better 

■ Solve math problems better 

■ Finish my homework more often 

■ Write better 

■ Use computers to do schoolwork better 
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Descriptive Statistics: 



Alpha 


Mean 


Standard 

Deviation 


Minimum 


25 th 

Percentile 


75 th 

Percentile 


Maximum 


0.87 


3.06 


0.74 


1 


2.71 


3.57 


4 
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Academic Motivation 



The Academic Motivation scale was computed to range from one to four, with four indicating 
that on average participants strongly agreed with the following statements: 

In general I... 



■ Try hard in school 

■ Pay attention in class 

■ Always come to class prepared 

■ Enjoy school 

■ Enjoy reading books for pleasure 

■ Enjoy math 

■ Enjoy writing 

■ Always finish my homework 

■ Do well in school 

Descriptive Statistics: 



Alpha 


Mean 


Standard 

Deviation 


Minimum 


25 th 

Percentile 


75 th 

Percentile 


Maximum 


0.86 


3.34 


0.59 


1 


3.00 


3.78 


4 




C-3 



Interactions with Staff 



The Interactions with Staff scale was computed to range from one to four, with four indicating 
that on average participants strongly agreed with the following statements: 

In this program. . . 

■ Staff treat me with respect 

■ I feel that I can talk to staff about things that are bothering me 

■ Staff really care about me 

■ Staff always keep their promises 

■ Staff care what I think 

■ Staff always try to be fair 

■ Staff think I can do things well 

■ Staff help me to try new things 

■ Staff think I can leam new things 

Descriptive Statistics: 



Alpha 


Mean 


Standard 

Deviation 


Minimum 


25 th 

Percentile 


75 th 

Percentile 


Maximum 


0.93 


3.35 


0.71 


1 


3.00 


4.00 


4 
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Exposure to New Experiences 



The Exposure to New Experiences scale was computed to range from one to four, with four 
indicating that on average participants strongly agreed with the following statements: 

In this program... 

■ I get a chance to do a lot of new things 

■ I get to do things that I don’t usually get to do anywhere else 

■ I get to work on projects that really make me think 

■ There is a lot for me to choose to do 

■ The activities really get me interested 

Descriptive Statistics: 



Alpha 


Mean 


Standard 

Deviation 


Minimum 


25 th 

Percentile 


75 th 

Percentile 


Maximum 


0.80 


3.20 


0.66 


1 


2.80 


3.80 


4 
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Sense of Belonging 



The Sense of Belonging scale was computed to range from one to four, with four indicating that 
on average participants strongly agreed with the following statements: 

In this program I feel like. . . 

■ I belong 

■ My ideas count 

■ lam successful 

■ This is a good place to hang out 

■ I matter 

■ I am safe 

Descriptive Statistics: 



Alpha 


Mean 


Standard 

Deviation 


Minimum 


25 th 

Percentile 


75 th 

Percentile 


Maximum 


0.86 


3.38 


0.64 


1 


3.00 


4.00 


4 




1.00 1.01 - 1.50 1.51 - 2.00 2.01 - 2.50 2.51 - 3.00 3.01 - 3.50 3.51 - 4.00 

Scale Score 
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Interactions with Peers 



The Interactions with Peers scale was computed to range from one to four, with four indicating 
that on average participants strongly agreed with the following statements: 

In this program I. . . 

■ Get to know other kids really well 

■ Can really trust the other kids 

■ Have a lot of friends 

■ Like the other kids 

■ Have a good time playing with other kids 

■ Get along with other kids 

Descriptive Statistics: 



Alpha 


Mean 


Standard 

Deviation 


Minimum 


25 th 

Percentile 


75 th 

Percentile 


Maximum 


0.84 


3.33 


0.61 


1 


3.00 


3.83 


4 
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Pro-social Behavior 



The Pro-social Behavior scale was computed to range from one to four, with four indicating that 
a participant engaged in the behavior at least six times over the past month, and one indicating 
that they never engaged in the behavior. The survey asked middle and high school participants 
to report on the following behaviors: 

In this program I. . . 

■ Helped someone stay out of a fight 

■ Told other students how I felt when they did something I liked 

■ Cooperated with others in completing a task 

■ Told other students how I felt when they upset me 

■ Protected someone from a bully 

■ Gave someone a compliment 

■ Helped other students solve a problem 

Descriptive Statistics: 



Alpha 


Mean 


Standard 

Deviation 


Minimum 


25 th 

Percentile 


75 th 

Percentile 


Maximum 


0.85 


2.50 


0.76 


1 


2.00 


3.00 


4 
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Appendix D 

Technical Properties of Observation Scales 



In Year 3, evaluators used PSA’s OST Observation Instrument to conduct an average of 12 
structured 15 minute observations in each of the 12 elementary- and middle-grades programs in the 
evaluation’s in-depth sample. (In Year 3, the evaluation used different methods for qualitative data 
collection in the three high school OST programs in the in-depth sample.) In total, 141 independent 
observations and 10 activity co-observations were conducted, with an average inter-rater reliability of 
0.76. Each observation indicator was rated from one to seven, with seven meaning that 
the indicator was highly evident and consistent throughout the observation. 

Skill-Building : The activity builds on and expands skills and content learned to increase youth 
knowledge and understanding. 

The Skill-Building scale combines ratings from the following indicators: 



■ Activity involves the practice or a progression of skills 

■ Staff challenges youth to move beyond their current level of competency 

■ Activity requires analytical thinking 

■ Staff employs varied teaching strategies 

■ Activity challenges students intellectually, creatively, developmentally, and/or 
physically 

■ Staff assists youth without taking control 

■ Staff verbally recognizes youth efforts and accomplishments 



Descriptive Statistics: 



Alpha 


Mean 


Standard 

Deviation 


Minimum 


25 th 

Percentile 


75 th 

Percentile 


Maximum 


0.88 


3.60 


1.41 


1.00 


2.42 


4.57 


6.71 




1.00 - 2.00 2.01 - 3.00 3.01 - 4.00 4.01 - 5.00 

Scale Score 



5.01 - 6.00 



6.01 - 7.00 



D-l 



Active Learning : The activity offers youth opportunities to actively participate in learning. 
The Active Learning scale combines ratings from the following indicators: 

■ Staff plan for and ask youth to work together 

■ Y outh are collaborative 

■ Youth take leadership responsibilities and roles. 

■ Y outh have opportunities to make meaningful choices 

■ Y outh assist one another 

■ Youth contribute opinions, ideas and concerns to discussions 

■ Staff encourages youth to share their ideas, opinions and concerns 

■ Staff asks youth to expand upon their answers and ideas 

Descriptive Statistics: 



Alpha 


Mean 


Standard 

Deviation 


Minimum 


25 th 

Percentile 


75 th 

Percentile 


Maximum 


0.67 


1.75 


0.80 


1.00 


1.12 


2.12 


5.00 




1.00 -2.00 2.01 - 3.00 3.01 - 4.00 4.01 -5.00 5.01 - 6.00 6.01 - 7.00 

Scale Score 



D-2 




Relationship-Focused : The activity focuses on developing positive relationships among youth 
and with staff. 

The Relationship-Focused scale combines ratings from the following indicators: 

■ Youth show positive affect to staff 

■ Youth are friendly and relaxed with one another 

■ Youth respect one another 

■ Staff shows positive affect toward youth 

■ Staff guides for positive peer interactions 

■ Staff uses positive behavior management techniques 

■ Staff is equitable and inclusive 

Descriptive Statistics: 



Alpha 


Mean 


Standard 

Deviation 


Minimum 


25 th 

Percentile 


75 th 

Percentile 


Maximum 


0.77 


4.21 


0.64 


2.00 


3.81 


4.50 


6.38 




Scale Score 



D-3 



Task-Oriented : The activity is organized with clear goals, and youth and staff are engaged and 
attentive. 

The Task-Oriented scale combines ratings from the following indicators: 

■ Activity is well organized 

■ Youth are on task 

■ Staff communicates goals, purposes, and expectations 

■ Youth listen actively and attentively to peers and staff 

■ Staff attentively listens to and observes youth 

Descriptive Statistics: 



Alpha 


Mean 


Standard 

Deviation 


Minimum 


25 th 

Percentile 


75 th 

Percentile 


Maximum 


0.81 


5.07 


0.82 


2.40 


4.60 


5.60 


6.80 
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Appendix E 

Comparison Groups for DOE Data Analyses 



To analyze the impact of OST participation on academic achievement and school 
engagement, the Y ear 3 evaluation employed a quasi-experimental design to compare 
participants with similar students who did not participate in an OST program. In creating a 
group of nonparticipants who were similar to participants along observable characteristics, the 
study attempted to separate the impact of OST participation on student outcomes from 
confounding factors such as gender, race, and family income. 

For each OST program, the evaluation identified a primary feeder school. Characteristics 
of the primary feeder school (e.g., proportion of ELL students and ELA/math results) were used 
to identify similar schools without OST or other city youth programs from which to select 
nonparticipants. The evaluation then matched a sample of students from the comparison schools 
with OST participants, based on the following demographic traits: grade in school, gender, race, 
free or reduced-price lunch eligibility, and ELL status. For two high school programs that 
attracted participants from across the city, similar nonparticipants were selected from schools in 
the same district without OST programs. Exhibit E-l indicates that the matching process 
produced a nonparticipant group similar to the OST participants on most observable 
characteristics. 



Exhibit E-1 

Characteristics of OST Participants and Matched Nonparticipants, in Percents 



Student Characteristics 


Participants 

(n=3,093) 


Nonparticipants 

(n=3,093) 


Difference 


P-value 


Gender (female) 


53.0 


52.1 


0.9 




English Language Learner 


17.7 


20.2 


2.5 




Free or Reduced-Price Lunch 


83.4 


86.6 


3.2 




Special Education / Related Services 


16.6 


17.0 


6.4 




Race 










African-American 


32.9 


30.5 


22.4 


0.05 


Asian 


16.6 


17.9 


1.3 


0.20 


Latino 


43.0 


44.6 


1.6 


0.21 


White 


6.9 


6.8 


0.1 


0.75 


Grade 










K to 4 


41.7 


39.4 


2.3 


0.08 


5 to 8 


36.1 


34.5 


1.6 


0.21 


9 to 12 


18.0 


17.2 


0.8 


0.44 


Other 


4.1 


8.8 


4.7 


0.00 



Exhibit reads: Fifty-three percent of OST participants in the sample were female, compared to 52 percent of 
nonparticipants. The difference was not statistically significant at the 0.05 level. 
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Appendix F 



Technical Properties of Correlations Between 
Program Quality and Youth Outcomes 

Information about the specific source and technical properties of each variable measuring 
program quality and youth outcomes is presented in Appendix G. In general, youth outcome 
variables are derived from participant survey data and from educational data from the New York 
City Department of Education (DOE). Program quality variables are generated from program 
director surveys, DYCD Online activity data, and aggregate participant survey data. 

Exhibit F-1 

Correlations between Range of Program Content and Youth Outcomes 
After 1 Year of OST Participation 



Youth Outcomes 


Spearman R 
Correlation 


P 


n 


Program Participation 


0.00 


0.94 


2761 


Sense of Belonging 


-0.21 


0.00 


1291 


Prosocial Behavior, 


0.01 


0.78 


750 


School Attendance 


-0.09 


0.00 


2857 


Academic Motivation 


-0.14 


0.01 


1295 


Academic Benefits 


-0.20 


0.00 


1273 


ELA Gains 2 


-0.04 


0.21 


799 


Math Gains 2 


0.00 


0.92 


901 


Total Credits Earned 3 


0.25 


0.00 


438 


, Middle and high school participants only 

2 Elementary and middle school participants only 

3 High school participants only 



Exhibit F-2 

Correlations between Exposure to New Experiences and Youth Outcomes 

After 1 Year of OST Participation 



Youth Outcomes 


Spearman R 
Correlation 


P 


n 


Program Participation 


-0.31 


0.00 


2865 


Sense of Belonging 


0.24 


0.00 


1354 


Prosocial Behavior, 


0.00 


0.89 


812 


School Attendance 


0.00 


0.74 


2958 


Academic Motivation 


0.08 


0.01 


1357 


Academic Benefits 


0.17 


0.00 


1336 


ELA Gains 2 


0.03 


0.35 


808 


Math Gains 2 


0.04 


0.27 


910 


Total Credits Earned 3 


-0.03 


0.46 


529 


, Middle and high school participants only 

2 Elementary and middle school participants only 

3 High school participants only 
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Exhibit F-3 

Correlations between Youth Interactions with Peers and Youth Outcomes 

After 1 Year of OST Participation 



Youth Outcomes 


Spearman R 
Correlation 


P 


n 


Program Participation 


-0.29 


0.00 


2865 


Sense of Belonging 


0.26 


0.00 


1354 


Prosocial Behavion 


0.06 


0.10 


812 


School Attendance 


-0.08 


0.00 


2958 


Academic Motivation 


0.09 


0.00 


1357 


Academic Benefits 


0.18 


0.00 


1336 


ELA Gains 2 


0.03 


0.34 


808 


Math Gains 2 


0.00 


0.97 


901 


Total Credits Earned 3 


-0.26 


0.00 


529 


1 Middle and high school participants only 

2 Elementary and middle school participants only 

3 High school participants only 



Exhibit F-4 

Correlations between Youth Interactions with Staff and Youth Outcomes 

After 1 Year of OST Participation 



Youth Outcomes 


Spearman R 
Correlation 


P 


n 


Program Participation 


-0.29 


0.00 


2865 


Sense of Belonging 


0.29 


0.00 


1354 


Prosocial Behavion 


0.07 


0.06 


812 


School Attendance 


-0.16 


0.00 


2958 


Academic Motivation 


0.07 


0.00 


1357 


Academic Benefits 


0.19 


0.00 


1336 


ELA Gains 2 


0.00 


0.99 


808 


Math Gains 2 


-0.04 


0.29 


910 


Total Credits Earned 3 


-0.03 


0.46 


529 


1 Middle and high school participants only 

2 Elementary and middle school participants only 

3 High school participants only 




















































































Exhibit F-5 

Correlations between Mix of Staff and Youth Outcomes 
After 1 Year of OST Participation 



Youth Outcomes 


Spearman R 
Correlation 


P 


n 


Program Participation 


0.01 


0.57 


2741 


Sense of Belonging 


0.11 


0.00 


1291 


Prosocial Behavior, 


0.10 


0.01 


812 


School Attendance 


-0.13 


0.00 


2839 


Academic Motivation 


0.07 


0.01 


1296 


Academic Benefits 


0.19 


0.00 


1275 


ELA Gains 2 


0.05 


0.20 


793 


Math Gains 2 


0.11 


0.00 


893 


Total Credits Earned 3 


0.27 


0.00 


529 


1 Middle and high school participants only 

2 Elementary and middle school participants only 

3 High school participants only 



Exhibit F-6 

Correlations between Staff Participation in Professional Development 
and Youth Outcomes After 1 Year of OST Participation 



Youth Outcomes 


Spearman R 
Correlation 


P 


n 


Program Participation 


-0.26 


0.00 


2520 


Sense of Belonging 


-0.10 


0.00 


1252 


Prosocial Behavior, 


0.00 


0.98 


812 


School Attendance 


0.00 


0.88 


2619 


Academic Motivation 


-0.13 


0.01 


1259 


Academic Benefits 


-0.08 


0.00 


1238 


ELA Gains 2 


0.01 


0.86 


749 


Math Gains 2 


-0.01 


0.75 


848 


Total Credits Earned 3 


0.27 


0.00 


529 


, Middle and high school participants only 

2 Elementary and Middle school participants only 

3 High school participants only 





















































































Exhibit F-7 

Correlations between Communication with Schools and Youth Outcomes 

After 1 Year of OST Participation 



Youth Outcomes 


Spearman R 
Correlation 


Q_ 


n 


Program Participation 


0.26 


0.00 


2741 


Sense of Belonging 


-0.10 


0.00 


1291 


Prosocial Behavion 


-0.08 


0.02 


812 


School Attendance 


-0.08 


0.00 


2839 


Academic Motivation 


-0.07 


0.02 


1296 


Academic Benefits 


0.07 


0.01 


1275 


ELA Gains 2 


-0.02 


0.66 


793 


Math Gains 2 


0.00 


0.99 


893 


Total Credits Earned 3 


-0.27 


0.00 


529 


1 Middle and high school participants only 

2 Elementary and middle school participants only 

3 High school participants only 



Exhibit F-8 

Correlations between Master Teacher and Youth Outcomes 
After 1 Year of OST Participation 



Youth Outcomes 


Point-Biserial 

Correlation 


P 


n 


Program Participation 


-0.17 


0.00 


2520 


Sense of Belonging 


0.22 


0.00 


1252 


Prosocial Behavion 


0.09 


0.02 


812 


School Attendance 


-0.18 


0.00 


2619 


Academic Motivation 


0.08 


0.01 


1259 


Academic Benefits 


0.19 


0.00 


1238 


ELA Gains 2 


0.03 


0.46 


749 


Math Gains 2 


0.10 


0.00 


848 


Total Credits Earned 3 


0.13 


0.00 


529 


1 Middle and high school participants only 

2 Elementary and middle school participants only 

3 High school participants only 
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Exhibit F-9 

Correlations between Communication with Parents and Youth Outcomes 

After 1 Year of OST Participation 



Youth Outcomes 


Spearman R 
Correlation 


P 


n 


Program Participation 


0.26 


0.00 


2741 


Sense of Belonging 


-0.14 


0.00 


1291 


Prosocial Behavior, 


-0.01 


0.73 


812 


School Attendance 


-0.11 


0.00 


2839 


Academic Motivation 


-0.10 


0.00 


1296 


Academic Benefits 


0.09 


0.00 


1275 


ELA Gains 2 


0.01 


0.89 


793 


Math Gains 2 


0.08 


0.02 


893 


Total Credits Earned 3 


0.13 


0.00 


529 


, Middle and high school participants only 

2 Elementary and middle school participants only 

3 High school participants only 



Exhibit F-10 

Correlations between Parent Liaison and Youth Outcomes 
After 1 Year of OST Participation 



Youth Outcomes 


Point-Biserial 

Correlation 


P 


n 


Program Participation 


0.24 


0.00 


2741 


Sense of Belonging 


-0.07 


0.02 


1291 


Prosocial Behavior, 


0.03 


0.40 


812 


School Attendance 


-0.11 


0.00 


2839 


Academic Motivation 


0.00 


0.95 


1296 


Academic Benefits 


0.19 


0.00 


1275 


ELA Gains 2 


0.04 


0.31 


793 


Math Gains 2 


0.01 


0.67 


893 


Total Credits Earned 3 


- 


- 


529 


, Middle and high school participants only 

2 Elementary and middle school participants only 

3 High school participants only 


















































































Exhibit F-11 

Correlations between Program Participation and Youth Outcomes 
After 1 Year of OST Participation 



Youth Outcomes 


Spearman R 
Correlation 


P 


n 


Program Participation 


- 


- 


- 


Sense of Belonging 


-0.05 


0.07 


1237 


Prosocial Behavior, 


0.04 


0.29 


738 


School Attendance 


0.17 


0.00 


2829 


Academic Motivation 


0.12 


0.00 


1239 


Academic Benefits 


0.04 


0.17 


1220 


ELA Gains 2 


0.00 


0.95 


773 


Math Gains 2 


0.05 


0.16 


869 


Total Credits Earned 3 


0.12 


0.00 


529 


, Middle and high school participants only 

2 Elementary and middle school participants only 

3 High school participants only 



Exhibit F-12 

Correlations between Range of Program Content and Youth Outcomes 
After 2 Years of OST Participation 



Youth Outcomes 


Spearman R 
Correlation 


P 


n 


Program Participation 


0.16 


0.00 


903 


Sense of Belonging 


-0.24 


0.00 


433 


School Attendance 


0.05 


0.11 


892 


Prosocial Behavior, 


0.01 


0.92 


183 


Academic Motivation 


-0.20 


0.01 


428 


Academic Benefits 


-0.23 


0.00 


420 


ELA Gains 2 


-0.06 


0.46 


181 


Math Gains 2 


0.00 


0.99 


208 


Total Credits 3 


0.24 


0.02 


103 


, Middle and high school participants only 

2 Elementary and middle school participants only 

3 High school participants only 





















































Exhibit F-13 

Correlations between Exposure to New Experiences and Youth Outcomes 

After 2 Years of OST Participation 



Youth Outcomes 


Spearman R 
Correlation 


P 


n 


Program Participation 


-0.43 


0.00 


929 


Sense of Belonging 


0.24 


0.00 


433 


Prosocial Behavion 


-0.02 


0.84 


183 


School Attendance 


0.11 


0.00 


918 


Academic Motivation 


0.15 


0.00 


428 


Academic Benefits 


0.23 


0.00 


420 


ELA Gains 2 


-0.13 


0.08 


184 


Math Gains 2 


-0.07 


0.28 


211 


Total Credits 3 


0.08 


0.41 


125 


1 Middle and high school participants only 

2 Elementary and middle school participants only 

3 High school participants only 



Exhibit F-14 

Correlations between Youth Interactions with Peers and Youth Outcomes 

After 2 Years of OST Participation 



Youth Outcomes 


Spearman R 
Correlation 


P 


n 


Program Participation 


-0.48 


0.00 


929 


Sense of Belonging 


0.31 


0.00 


433 


Prosocial Behavior! 


-0.01 


0.87 


183 


School Attendance 


-0.02 


0.50 


918 


Academic Motivation 


0.18 


0.00 


428 


Academic Benefits 


0.24 


0.00 


420 


ELA Gains 2 


-0.35 


0.00 


184 


Math Gains 2 


-0.13 


0.07 


211 


Total Credits 3 


-0.11 


0.24 


125 


1 Middle and high school participants only 

2 Elementary and middle school participants only 

3 High school participants only 
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Exhibit F-15 

Correlations between Youth Interactions with Staff and Youth Outcomes 

After 2 Years of OST Participation 



Youth Outcomes 


Spearman R 
Correlation 


P 


n 


Program Participation 


-0.43 


0.00 


929 


Sense of Belonging 


0.35 


0.00 


433 


Prosocial Behavior, 


-0.01 


0.90 


183 


School Attendance 


-0.12 


0.00 


918 


Academic Motivation 


0.17 


0.00 


428 


Academic Benefits 


0.26 


0.00 


420 


ELA Gains 2 


-0.22 


0.00 


184 


Math Gains 2 


-0.04 


0.58 


211 


Total Credits 3 


-0.08 


0.41 


125 


1 Middle and high school participants only 

2 Elementary and middle school participants only 

3 High school participants only 



Exhibit F-16 

Correlations between Mix of Staff and Youth Outcomes 
After 2 Years of OST Participation 



Youth Outcomes 


Spearman R 
Correlation 


P 


n 


Program Participation 


-0.15 


0.00 


879 


Sense of Belonging 


0.14 


0.04 


415 


Prosocial Behavior, 


0.07 


0.33 


183 


School Attendance 


-0.16 


0.00 


870 


Academic Motivation 


0.11 


0.03 


413 


Academic Benefits 


0.09 


0.08 


404 


ELA Gains 2 


-0.28 


0.00 


184 


Math Gains 2 


0.03 


0.66 


210 


Total Credits3 3 


0.11 


0.23 


125 


, Middle and high school participants only 

2 Elementary and middle school participants only 

3 High school participants only 




















































































Exhibit F-17 

Correlations between Staff Participation in Professional Development 
and Youth Outcomes After 2 Years of OST Participation 



Youth Outcomes 


Spearman R 
Correlation 


P 


n 


Program Participation 


-0.18 


0.00 


802 


Sense of Belonging 


-0.18 


0.00 


395 


Prosocial Behavion 


0.05 


0.50 


183 


School Attendance 


-0.12 


0.00 


793 


Academic Motivation 


0.12 


0.02 


394 


Academic Benefits 


-0.22 


0.00 


386 


ELA Gains 2 


0.11 


0.16 


168 


Math Gains 2 


0.02 


0.76 


194 


Total Credits 3 


0.11 


0.23 


125 


1 Middle and high school participants only 

2 Elementary and middle school participants only 

3 High school participants only 



Exhibit F-18 

Correlations between Communication with Schools and Youth Outcomes 

After 2 Years of OST Participation 



Youth Outcomes 


Spearman R 
Correlation 


P 


n 


Program Participation 


0.33 


0.00 


879 


Sense of Belonging 


-0.08 


0.11 


415 


Prosocial Behavior! 


0.02 


0.77 


183 


School Attendance 


-0.17 


0.00 


870 


Academic Motivation 


-0.06 


0.25 


413 


Academic Benefits 


-0.01 


0.84 


404 


ELA Gains 2 


-0.14 


0.06 


184 


Math Gains 2 


-0.06 


0.35 


210 


Total Credits 3 


-0.11 


0.24 


125 


1 Middle and high school participants only 

2 Elementary and middle school participants only 

3 High school participants only 



















































































Exhibit F-19 

Correlations between Master Teacher and Youth Outcomes 
After 2 Years of OST Participation 



Youth Outcomes 


Point-Biserial 

Correlation 


P 


n 


Program Participation 


-0.29 


0.00 


802 


Sense of Belonging 


0.22 


0.00 


395 


Prosocial Behavion 


0.01 


0.94 


183 


School Attendance 


-0.11 


0.00 


793 


Academic Motivation 


-0.00 


0.98 


394 


Academic Benefits 


0.17 


0.00 


386 


ELA Gains 2 


-0.25 


0.00 


168 


Math Gains 2 


-0.08 


0.30 


194 


Total Credits 3 


0.00 


0.99 


125 


1 Middle and high school participants only 

2 Elementary and middle school participants only 

3 High school participants only 



Exhibit F-20 

Correlations between Communication with Parents and Youth Outcomes 

After 2 Years of OST Participation 



Youth Outcomes 


Spearman R 
Correlation 


P 


n 


Program Participation 


0.37 


0.00 


879 


Sense of Belonging 


-0.21 


0.00 


415 


Prosocial Behavion 


-0.05 


0.47 


183 


School Attendance 


-0.17 


0.00 


870 


Academic Motivation 


-0.16 


0.00 


413 


Academic Benefits 


-0.13 


0.01 


404 


ELA Gains 2 


-0.29 


0.00 


184 


Math Gains 2 


-0.06 


0.43 


210 


Total Credits 3 


0.00 


0.99 


125 


1 Middle and high school participants only 

2 Elementary and middle school participants only 

3 High school participants only 
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Exhibit F-21 

Correlations between Parent Liaison and Youth Outcomes 
After 2 Years of OST Participation 



Youth Outcomes 


Point-Biserial 

Correlation 


P 


n 


Program Participation 


0.25 


0.00 


879 


Sense of Belonging 


-0.16 


0.00 


415 


Prosocial Behavior, 


0.03 


0.72 


183 


School Attendance 


-0.07 


0.04 


870 


Academic Motivation 


-0.22 


0.00 


413 


Academic Benefits 


-0.06 


0.25 


404 


ELA Gains 2 


-0.31 


0.00 


184 


Math Gains 2 


0.01 


0.90 


210 


Total Credits 3 


- 


- 


125 


1 Middle and high school participants only 

2 Elementary and middle school participants only 

3 High school participants only 



Exhibit F-22 

Correlations between Program Participation and Youth Outcomes 
After 2 Years of OST Participation 



Youth Outcomes 


Spearman R 
Correlation 


P 


n 


Program Participation 


- 


- 


- 


Sense of Belonging 


-0.12 


0.02 


433 


Prosocial Behavior, 


0.03 


0.74 


183 


School Attendance 


0.13 


0.00 


916 


Academic Motivation 


-0.03 


0.50 


428 


Academic Benefits 


-0.04 


0.39 


420 


ELA Gains 2 


0.20 


0.01 


182 


Math Gains 2 


0.07 


0.32 


209 


Total Credits 3 


0.04 


0.70 


125 


, Middle and high school participants only 

2 Elementary and middle school participants only 

3 High school participants only 
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Exhibit F-23 

Correlations between Range of Program Content and Youth Outcomes 
After 3 Years of OST Participation 



Youth Outcomes 


Spearman R 
Correlation 


P 


n 


Program Participation 


0.32 


0.00 


215 


Sense of Belonging 


-0.10 


0.27 


116 


School Attendance 


-0.07 


0.33 


212 


Academic Motivation 


-0.05 


0.60 


119 


Academic Benefits 


-0.11 


0.24 


114 


ELA Gains 2 


0.07 


0.75 


24 


Math Gains 2 


-0.15 


0.45 


28 


2 Elementary and middle school participants only 



Exhibit F-24 

Correlations between Exposure to New Experiences and Youth Outcomes 

After 3 Years of OST Participation 



Youth Outcomes 


Spearman R 
Correlation 


P 


n 


Program Participation 


-0.18 


0.01 


217 


Sense of Belonging 


0.26 


0.01 


116 


School Attendance 


0.22 


0.00 


214 


Academic Motivation 


0.02 


0.88 


119 


Academic Benefits 


0.12 


0.21 


114 


ELA Gains 2 


0.26 


0.22 


24 


Math Gains 2 


-0.23 


0.24 


28 


2 Elementary and middle school participants only 
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Exhibit F-25 

Correlations between Youth Interactions with Peers and Youth Outcomes 

After 3 Years of OST Participation 



Youth Outcomes 


Spearman R 
Correlation 


P 


n 


Program Participation 


-0.45 


0.00 


217 


Sense of Belonging 


0.22 


0.02 


116 


School Attendance 


0.01 


0.93 


214 


Academic Motivation 


0.03 


0.71 


119 


Academic Benefits 


0.07 


0.49 


114 


ELA Gains 2 


-0.09 


0.69 


24 


Math Gains 2 


-0.19 


0.34 


28 


2 Elementary and middle school participants only 



Exhibit F-26 

Correlations between Youth Interactions with Staff and Youth Outcomes 

After 3 Years of OST Participation 



Youth Outcomes 


Spearman R 
Correlation 


P 


n 


Program Participation 


-0.21 


0.00 


217 


Sense of Belonging 


0.29 


0.00 


116 


School Attendance 


-0.09 


0.21 


214 
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Exhibit F-27 

Correlations between Mix of Staff and Youth Outcomes 
After 3 Years of OST Participation 
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Exhibit F-28 

Correlations between Staff Participation in Professional Development 
and Youth Outcomes After 3 Years of OST Participation 
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Exhibit F-29 

Correlations between Communication with Schools and Youth Outcomes 

After 3 Years of OST Participation 
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2 Elementary and middle school participants only 



Exhibit F-30 

Correlations between Master Teacher and Youth Outcomes After 3 Years of OST 

Participation 



Youth Outcomes 


Point-Biserial 

Correlation 


P 


n 


Program Participation 


-0.14 


0.06 


185 


Sense of Belonging 


0.25 


0.01 


105 


School Attendance 


-0.09 


0.22 


184 


Academic Motivation 


-0.16 


0.09 


108 


Academic Benefits 


0.15 


0.14 


103 


ELA Gains 2 


-0.08 


0.70 


24 


Math Gains 2 


0.18 


0.36 


28 


2 Elementary and middle school participants only 
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Exhibit F-31 

Correlations between Communication with Parents and Youth Outcomes 

After 3 Years of OST Participation 
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2 Elementary and middle school participants only 



Exhibit F-32 

Correlations between Parent Liaison and Youth Outcomes 
After 3 Years of OST Participation 
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2 Elementary and middle school participants only 



Exhibit F-33 

Correlations between Program Participation and Youth Outcomes 
After 3 Years of OST Participation 
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Appendix G 

Technical Properties of Measures of Program Quality 



Variable Name 


Description 


From Participant Su 


irvey 



A continuous scale variable, computed by averaging responses to the following items: 



In this program... 

I get a chance to do a lot of new things 
I get to do things that I don’t usually get to do anywhere else 
I get to work on projects that really make me think 
There is a lot for me to choose to do 
The activities really get me interested 

Response categories were: agree a lot (4), agree a little (3), disagree a little (2), and disagree a lot (1). Scale 
scores were averaged to compute an aggregated scale score for each program. 

M=3.21, SD=0.29, Range=1-4, n=87 

A continuous scale variable, computed by averaging responses to the following items: 

In this program... 

Staff treat me with respect 

I feel that I can talk to staff about things that are bothering me 

Staff really care about me 

Staff always keep their promises 

Staff care what I think 

Staff always try to be fair 

Staff think I can do things well 

Staff help me to try new things 

Staff think I can learn new things 

Response categories were: agree a lot (4), agree a little (3), disagree a little (2), and disagree a lot (1). Scale 
scores were averaged to compute an aggregated scale score for each program. 

M=3.38, SD=0.33, Range=1-4, n=87 

A continuous scale variable, computed by averaging responses to the following items: 

In this program I... 

Get to know other kids really well 
Can really trust the other kids 
Have a lot of friends 
Like the other kids 

Have a good time playing with other kids 
Get along with other kids 

Response categories were: agree a lot (4), agree a little (3), disagree a little (2), and disagree a lot (1). Scale 
scores were averaged to compute an aggregated scale score for each program. 

M=3.32, SD=0.21, Range=1-4, n=87 



Interactions with 
Peers 



Interactions with 
Staff 



Exposure to New 
Experiences 



G-l 








From Program Director Survey 

A continuous scale variable, computed by averaging directors' responses to the following items: 

How often do you discuss the following topics with the principal(s), teachers, or 
other key staff from the school(s) that your participants attend? 

Planning OST program content 
Curriculum concepts currently being taught in school 
Homework assignments 
The needs or progress of individual students 
Issues related to classrooms/sharing space 
School Communication Student discipline policies 

OST program staffing 

OST recruiting/ enrollment policies (e.g., targeting students) 

State and local standards in reading, math, and/or science 



Response categories were: “at least 2 to 3 times a month” (5), “once a month” (4), “1 to 2 times a semester” (3), “1 to 
2 times a year" (2), “never” (1). 

M=3.05, SD=1. 12, Range=1 - 5, n=87 



Partnership with 
Families 



Communication with 
Parents 



Professional 
Development 
Opportunities for Staff 



A dichotomous variable indicating whether or not a program has a parent liaison. 

M=0.45, SD=0.50, Range=0 - 1, n=87 

A continuous scale variable, computed by averaging directors’ responses to the following items: 

How often do you. . . 

Send materials about the program home to parents 

Hold events or meetings to which parents are invited 

Hold events or meetings to which community members are invited 

Have conversations with parents over the phone 

Meet with one or more parents 

Response categories were: “at least 4 to 5 days a week” (4), “about 1 to 3 days a week” (3), “a few times a month” (2), 
“less than once a month” (1). 

M=1.91, SD=0.49, Range=1 -4, n=87 

An index computed by totaling the number of technical assistance tools or professional development opportunities 
that a program's staff used or participated in. 

In the current year (2007-08), what technical assistance tools or professional 
development opportunities have YOUR STAFF used or participated in? 

NYSAN Quality Self-Assessment tool 

Internal staff orientation 

Staff meetings at the program 

Staff meetings at the provider organization 

Coaching/mentoring opportunities 

Off-site workshops 

Institutes or conferences 

On-site consultations by PASE or other TA providers 
Other trainings not directly related to OST 



Master Teacher 



Mix of Staff 



M=3.85, SD=1.79, Range=1 -9, n=87 

A dichotomous variable indicating whether or not a program has a master teacher. 
M=0.51, SD=0.50, Range=0 - 1, n=87 

An index computed by counting the number of types of staff a program employs. 

Types of staff include: 

School administrators 
Certified teachers 
College students 
Teen staff 
Specialists 

Other adults with a college degree 
Other adults without a college degree 

M=4.52, SD=1.61, Range=1 - 7, n=87 
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From DYCD Online 



An index computed by counting the number of activity categories in which a program offered 10 or more hours of 
programming. 



Program Content 
Diversity 



The six collapsed categories include: 

Academic enhancement: academic enhancement, homework help, computer instruction, literacy, and numeracy 
Arts and culture: arts and culture 

Recreation: recreation, unstructured physical recreation, and structured physical recreation 
Career and work: career awareness and school to work 
Community building: community building 
Life skills: life skills and financial literacy 



1/1=4.10, SD=1. 12, Range= 1 - 6, n=87 
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